Risk-Based Computer System Validation and Rational Testing

Introduction

Risk based approaches to validation of computerized systems have been heavily promoted since the publication of GAMP 5 and ASTM E2500.  Yet we continue to see examples of validation overkill in the industry.  A lack of risk analysis can result in testing activities far exceeding the need to demonstrate system reliability, reproducibility, and fit for intended use.

Background and Verification Requirements

The following example involves the need to validate a set of environmental monitoring particle counters.  The instruments are designed to measure counts and sizes of non-viable particles in a continuous stream of sampled air.  The particle counters can be configured to sound and display audible and visible alarms if particle limits are exceeded or if air flow rates fall above or below established limits.   Multiple individual units are to be qualified, all from the same vendor, covering three different, but similar models of equipment.  The following comments apply only to validation of the software components of the system, and assume that each instrument is subject to standard calibration to ensure reliability of the sampling and analysis hardware components.

Not-to-exceed particulate alarm limits are to be programmed into the units, covering seven different room classifications, each room with its own particulate limits.  In addition alarm limits are to be programmed for low and high air flow rates to ensure proper sampling of particulate counts.   Traditional qualification approaches might dictate that each unit be fully tested for each programmed particulate and flow rate alarm limit.

Impacts of Validation Approach

The impact of such a full testing approach, versus a more risk-based approach, can be significant.   By evaluating the overall risk of testing scenarios efficiencies in testing can be achieved without risk to overall compliance.  Basic assumptions are evaluated in building a rational justification for eliminating redundant verifications, and reducing time and effort required to demonstrate that the monitoring units have been configured properly and reliably perform their functions.

For example, confirmation of software/firmware versions can be used to justify limited functional testing on each independent unit, assuming some verification of vendor quality systems has been done.  By performing more extensive functional testing on the first unit of each instrument model, verifying operational alarms for each specific configuration, and then confirming identical software versions and configurations on subsequent units, the remaining units could be qualified with basic operational tests to confirm functioning of the physical sampling and alarm components.

Risk-based Rationale

Additional risk-based assumptions could be used to further reduce testing requirements.  For example, by focusing on selected alarm ranges, the initial functional testing might evaluate low, medium, and high particulate counts to establish range of operation of the units.    This could replace testing for all seven combinations of alarm limits configured in each unit.

Since the software simply compares a cumulative particle count against a configured limit, it should not be necessary to verify every potential limit setting to ensure that alarms activate.  In fact this is not actually feasible. Limit settings, once established in the software configuration, are fixed and are not modified. As a result verification of the alarms using the lowest count configuration and the highest count configuration (and possibly a mid-range value) would verify limit settings, alarm activities and system functionality for all alarm settings. System functionality is not based on specific room requirements, but rather on limit settings entered for each program.

Summary

The result is a significant reduction in the volume of test documentation that must be generated, and time and effort required to complete verification activities, while still maintaining a defendable rationale for regulatory compliance.  In approaching any computer system validation effort, it is important to evaluate aspects of each system that may or may not introduce sources of variation.  This can be used to establish a rational approach to testing and verification that will adequately demonstrate the system’s intended use and reliability, but not introduce unnecessary levels of verification that do not add value to the process or outcome.