By: Wade Yandell and Dave Wexler

 

IVD manufacturers and life science and clinical laboratories have benefited from the recent technological advances in liquid handling (e.g., handheld pipettes, automated liquid handlers). However, since automated liquid handling is a relatively new technology, standardized performance assessment and calibration guidelines do not currently exist. The consequences of poor performance due to liquid handling errors can be severe, such as inefficiency, failed experiments, and the release of inaccurate test results.

A critical component in a quality control program for automated liquid handlers is obtaining data regarding each lab instrument's performance. Such data allow laboratories to take corrective actions before an instrument is out of specification, and ensure maximum efficiency and accurate and precise results.

Liquid handling accuracy and precision are also critical for molecular diagnostic companies such as XDx (Brisbane, CA) that focus on gene expression testing. The company's AlloMap molecular expression testing is an in vitro diagnostic multivariate index assay test service, performed in a single laboratory, assessing the gene expression profile of RNA isolated from peripheral blood mononuclear cells. AlloMap testing is intended to aid in the identification of heart transplant recipients with stable allograft function. Such recipients have a low probability of moderate or severe acute cellular rejection at the time of testing in conjunction with standard clinical assessment.

Automated liquid handlers are critical to not only the successful production of the AlloMap qRT-PCR assay plates but also the preparation of patient samples for proper testing. The steps, including normalizing RNA mass for cDNA synthesis and setting up the qRT-PCR reactions, all involve automated liquid handling. In addition, because these assays are volume-dependent, inaccurate volume transfers can affect the test results. Due to the small liquid volumes typically handled at XDx (i.e., 5 µl on average), inaccuracies of even a fraction of one microliter can affect results and impede efficiency. The consequences of liquid handling errors include inaccurate test results and products that fail the quality control process, leading to wasted time and materials, inefficiency, and unnecessary costs. For example, the universal master mix can cost $20,000 for a set of 300 qRT-PCR plates for testing.

As part of XDx's master calibration and validation plans, and to avoid the costs and consequences of liquid handling errors, the company maintains a rigorous liquid handling performance monitoring process that produces a plethora of instrument accuracy and precision data (see Figure 1). Collectively, the data provide significant insight into the need for routine performance assessments of critical automated liquid handling transfers, especially on a tip-by-tip basis. This article discusses the longitudinal data (data showing performance over time) collected between January 2006 and March 2008. It also offers guidance on optimization practices and the frequency between calibration intervals for automated liquid handlers.

Study Parameters

The data included in XDx's liquid handling longitudinal study were compiled using both gravimetric and photometric calibration methods. Prior to July 2007, the company employed gravimetry as a standard method of performance assessment in which an analytical balance was used to weigh liquid volumes. After July 2007, XDx assessed liquid handling instruments with the MVS Multichannel Verification System by Artel (Westbrook, ME). That system is based on dual-dye ratiometric photometry and provides tip-by-tip accuracy and precision data for transfers into microtiter plates.

The study included performance data collected from the following liquid handling instruments: four Biomek 2000 instruments, one Biomek 3000 instrument (grouped with the Biomek 2000 instruments for data analysis purposes), and five Biomek FXp Span-8 systems (with 1000-µl syringes) by Beckman Coulter Inc. (Fullerton, CA) (see Table I).

The data from the Biomek 2000 and 3000 liquid handlers were analyzed according to tool, which is a term for different interchangeable pipetting heads that are used depending on desired pipetting function and/or volume. In addition, the data from the Biomek FXp instruments were analyzed according to pipetting technique. Each technique has different defined pipetting parameters for the transfer of liquid. Examples of such parameters include but are not limited to aspirate height, dispense height, speed, and trailing air gap. The data were grouped by liquid handling platform (i.e., Biomek 2000/3000 and Biomek FXp) and by tool or technique type, where each tool and technique had a discrete set of test volumes. Table I lists the liquid handling tools and techniques included in the study.

The Biomek 2000 and 3000 are volumetrically verified on a monthly basis at XDx, and the Biomek FXp instruments are verified twice per year. Deviations from the documented frequency of maintenance can occur when an instrument is not in service for an extended period of time or when monitoring of high-usage instruments is used to generate data for longitudinal analysis (e.g., the Biomek FXp instruments were checked quarterly during this study). The performance of each instrument is qualified according to internal accuracy and precision specifications. Such specifications were defined according to specific acceptance criteria required for applications at XDx.

Liquid Handling Optimization

After analyzing the longitudinal data, several conclusions were made regarding the optimal use of automated liquid handlers and performance monitoring practices. From these conclusions, several guidelines for optimal liquid handling practices were established.

Systematic or Random Errors

The first step in optimizing liquid handling operations is to determine whether an error that exists is random or systematic. Systematic or consistent variations indicate that the liquid handler requires maintenance due to instrument issues, such as faulty or damaged O-rings, tubing problems, or motors and pumps that have lost their home positions. Conversely, random failures are nonsystematic, are generally not instrument-specific, and require a different course for corrective action. To identify, account for, and mitigate random errors, regular sampling, performance assessment, and calibration are required.

In analyzing XDx's longitudinal performance data, systematic changes were not common. Rather, the majority of the variations in liquid handler performance were random. The data indicated that while serious maintenance was not often required, regular calibration was necessary to identify and correct the random variations.

Considering Calibration Processes

When developing programs (called “methods”) for automated liquid handling operations, IVD manufacturers should ensure that performance assessment and calibration processes can be implemented to verify an instrument's performance when the assay-defined techniques and methods are applied. Pipetting parameters for a given method or assay should be predefined and should not be deviated from. When pipetting parameters are continuously modified and customized for a particular assay, having a true determination of liquid handler performance becomes increasingly difficult, due to routine method manipulations.

Mimicking Assay Parameters

The parameters tested during the liquid handler verification process should mimic the parameters used during normal liquid handling operations. Therefore, volume verification processes should use the same combination of tip type, liquid type, and liquid reservoir as the actual assay. Factors such as liquid handling mode (forward versus reverse), dispense (wet or dry), microtiter plate geometry, aspirate and dispense speed, and air gap usage can influence volume transfer accuracy and precision. When it is feasible, the pipetting technique should be mimicked during the calibration process to provide a clear indication of actual instrument performance under assay conditions.

Choosing Appropriate Calibration Frequency

To minimize laboratory errors, automated liquid handler performance should be verified at shorter intervals than the average time it takes for the pipetting steps to fall out of specifications. Historical pipetting data can indicate an instrument's propensity for error over time and reveal the appropriate frequency for calibration. The calibration frequency must be balanced against the cost of calibrating and the time required for a technical expert to perform the volumetric verification. While calibrating too frequently can waste time, effort, and money, calibrating too infrequently could allow the instrumentation to operate outside of specification for a specific protocol and affect data quality.

In launching a calibration plan, a conservative approach would be to verify calibration two or three times as frequently as the liquid handler manufacturer specifies. As data are acquired on an instrument's performance, the calibration frequency can be adjusted accordingly. Longitudinal pipetting data continually help to refine calibration schedules and tolerances for the liquid handling instruments.

Verifying Individual Channel Performance

While the average dispense for all channels or tips of an automated liquid handler will often be within the performance specifications, one or more individual channels may not meet the performance criteria. Figure 2 shows varying channel performance information when the data are analyzed on a tip-by-tip basis.

In Figure 2, channel two consistently failed to meet the acceptance criteria. After several runs, it was evident that this channel was not functioning properly. In this case, the syringe pump in a Biomek FXp instrument that controlled channel two was replaced. Following repair and calibration, the final run data showed that all channels were able to pass internal performance specifications (see Figure 2).

Figure 3 demonstrates another example highlighting the importance of individual channel performance. This figure shows the performance of the mean over three replicates for the Biomek FXp-1 instrument at one calibration interval using the Span-8 2-10 technique. Instrument performance data are often analyzed per system and per calibration interval. This instrument's average transferred volume for all channels falls within the specifications, and does not indicate an issue with the liquid handler.

However, when evaluating the tip-by-tip performance for the average data, multiple channels of this liquid handler (channels 5, 7, and 8) fall outside the defined specifications (see Figure 3). This critical information is not discovered when only an average of the eight channels is analyzed.

The channel-by-channel analysis was possible by using a calibration method for capturing tip-by-tip data for every volume transfer (see Figure 3). Alternatively, calibration methods that only provide aggregate information on the accuracy and precision of all channels in a liquid handler (e.g., gravimetric methods) fail to offer the same critical information needed to ensure the proper performance of all liquid handling steps. As a result of only using aggregate information, subsequent assays performed with poor performing tips might be unknowingly compromised. The data indicate that the overall average dispense should not be relied on to reflect the accuracy of all individual channels. For data quality, each channel and the average of all channels of an automated liquid handler must meet performance specifications.

Obtaining Accuracy and Precision Information

The precision and accuracy of automated liquid handlers must be assessed to get a complete sense of an automated liquid handler's performance. Data from XDx's longitudinal study indicate several instances in which an automated liquid handling tool or technique is precise and inaccurate, or accurate and imprecise (see Figure 4, which shows the average volume values during replicate trials).

The accuracy of a group of repeated or replicate measurements can be determined by calculating the mean of the group and comparing that average value with the target value. The accuracy for a group of volume measurements refers to the deviation of the group's mean value from the target volume.

Precision indicates how close a group of measurements are to one another. The tighter the data replicates, the more predictable future results will be. For this reason, good precision has a predictive value and gives confidence in future results. A precise or closely clustered data set has a smaller coefficient of variation (CV) and is more reliable than one that is widely scattered.

Most automated liquid handler manufacturers only specify precision performance. However, data from an instrument that performs precisely but not accurately is indicated in the right side of Figure 4. According to precision-only specifications, the instrument would appear to be performing well even though almost all of the volume dispenses fell outside of the designated tolerance range for accuracy.

In addition, knowing accuracy alone is of limited use. Data on the left side of Figure 4 show that the eight replicate measurements averaged 10 µl. However, predicting how likely the next dispense will be within the specified limits is impossible. The automated liquid handler might deliver 9.9, 10.0, and 10.1 µl, while another might deliver 8.0, 10.0, and 12.0. The averages of both sets of data are 10 µl, and both are perfectly accurate. However, for volume-dependent protocols, the first data set is preferred.

Troubleshooting and Adjusting Performance Specifications

By using longitudinal data, laboratories might be able to tighten certain pipetting specifications to improve their automated liquid handlers' performance. However, it is important not to exceed the liquid handler's ability when tightening specifications.

For example, in certain high-volume transfers, little variation has been observed over time, allowing pipetting specifications to be tightened to decrease the potential variability of assay performance. In Figure 5, target volumes of 700 and 900 µl ±7% could be tightened to ±3%, leading to better reliability for the method.

In another example, XDx applied a serial dilution protocol (Biomek 2000 with the P1000 tool) using a concentrated template. By evaluating the longitudinal data, it was determined that the accuracy specification of each volume transfer could be tightened from 6 to 3% without the instrument going out of tolerance. The tightened pipetting specifications resulted in improved efficiency and better performance of the method using the P1000 protocol, serving to reduce waste in terms of dead volume without extra time or labor hours required.

If operations are not adversely affected, laboratories could benefit by loosening certain pipetting specifications. Such loosening reduces the burden on technical experts performing the volumetric verification in assessing the automated liquid handlers' performance. Certain tools and techniques have traditionally been very problematic due to their high variance and frequency of drifting outside the specifications. After conducting studies to determine confidence intervals around expected values, or guard band studies, adopting looser specifications for particularly troublesome pipetting steps is possible.

Historical performance data also provide input to laboratory managers to assist in troubleshooting assays that produce incorrect results. Without historical data, ruling out the instrument as the root cause of the questionable data is difficult and time consuming, and requires a lengthy calibration process (4-6 hours). Alternatively, when historical data are available, the instrument has already been characterized and only a quick spot check needs to be conducted (requiring less than 30 minutes) to eliminate the instrument as the error source.

For example, in Figure 6, the March 2008 data points for two different techniques of the FXp-1 automated liquid handler are all within the acceptable range but are on the low end of specifications for the 5-µl and 10-µl target volume transfers. In the case in which a liquid handling method uses pipetting steps with these two techniques, which are all calibrated on the low end of the acceptable accuracy range, the cumulative effect could be detrimental to the final test results. In this example, the method was not designed with the specifications of the volume transfers in mind, and therefore longitudinal data helped to identify the root cause of the errors.

Conclusion

A number of steps can be implemented to optimize performance monitoring of automated liquid handlers. Critical to the process is the consistent collection of tip-by-tip performance data on automated liquid handlers. Such information can shape quality control practices and improve confidence in the data that are produced using automated liquid handling instruments. If they are implemented properly, such suggestions can improve the robustness of laboratory operations and reduce the risks to patient health in the clinical setting.