By: Clark A. Rundell

Furthermore, in the May 2010 Agency for Healthcare Research and Quality (AHRQ) report, “Quality, Regulation and Clinical Utility of Laboratory-developed Molecular Tests,” an international summary of proficiency test results showed widely varying levels of performance, with one survey reporting results of 28 to 98% of all results correct, and another survey reporting 94 to 100% of all results correct.¹ This report strongly indicates that QC practice is below standard in clinical laboratories just as it is in DTC companies.

The report was the result of a request to AHRQ, the health services research arm of HHS, from CMS, and it draws data from an extensive review of literature, informational Web pages from laboratories and professional societies, and information available from government agencies. Published proficiency results from seven organizations were summarized. The surveys comprised PCR-based tests, fluorescence in situ hybridization, PCR in general, and sequencing. Four of the organizations’ results were based on one or two surveys, and three results were based on seven or eight surveys. Overall, 85 to 95% correct results was a common finding, and the authors concluded that quality is improving, as indicated by the most recent surveys. However, findings as low as 28% and 44% were reported; more proficiency data is needed for accurate assessment.

Since proficiency testing is the standard measure of laboratory performance used by regulatory agencies, these results are viewed as evidence of the need for increased quality in molecular diagnostics. Proficiency data are not perfect; some “erroneous” results may be due to problems with proficiency samples themselves. However, molecular proficiency programs currently test only a small percentage of the available molecular tests, and these are often the highest volume, most robust tests. Even a relatively low level of error seen in current proficiency programs suggests that accuracy concerns are warranted. 

There is no current consensus on the allowable medical error for molecular tests.  But while experience shows that lab tests with zero error do not exist, and that in most cases some error is medically allowable, most experts would agree that even the 15% error rate seen in a few of the best performing surveys summarized in the AHRQ proficiency report is unacceptable for a once-in-a-lifetime test or tests used to direct treatment for life-threatening diseases such as cancer. Evidence exists suggesting that the error rate is higher in tests for rare mutations not typically found in proficiency samples.^2-4

Although it is clear that molecular testing is complex and requires stringent adherence to good quality control practices, even experienced and prestigious laboratories have had molecular test quality problems with subsequent regulatory action and negative publicity. In one case, apparently incorrect results were reported for several patients; inspectors attributed the errors to quality control procedures not being followed properly.⁵ And the regulatory community is responding to evidence of inadequate quality of molecular tests.

In the United States, molecular testing is regulated by CMS through the Clinical Laboratory Improvement Amendments of 1988 (CLIA) and by FDA. CLIA regulates laboratories performing the tests, while FDA regulates laboratory tests, having reviewed and cleared approximately 145 molecular tests as of this writing. Although hard evidence is lacking, the stringency of the Quality System Regulation (21 CFR Part 820) and FDA submission requirements lends credibility to the assumption that FDA-approved or -cleared tests may be more reliable than tests not manufactured according to GMP. The AHRQ report suggests that there are more than 1400 molecular tests that are laboratory-developed tests (LDTs), which are not currently FDA regulated, while information on the GeneTests Web site suggests that the number may be greater than 2000.⁶ Historically, FDA has exercised “enforcement discretion” for oversight of LDTs, allowing the labs to develop and validate them according to CLIA high-complexity regulations without FDA review. However, at the July 19, 2010 FDA Public Meeting on Oversight of Laboratory-Developed Tests, FDA announced that it had “decided to exercise authority over LDTs.”⁷ It’s not clear yet how FDA will proceed, but some surmise that particularly high-risk molecular tests, such as those for oncology, will be reviewed first. 

FDA’s first action on LDTs—regulation of the complex tests known as In Vitro Diagnostic Multivariate Index Assays (IVDMIA)—will be subsumed into a new, more comprehensive guidance for regulation of LDTs. As an indication of the seriousness of its intent, FDA sent 19 sanction letters to labs offering LDT tests between July 12 and July 19, 2010. Understandably, laboratorians are concerned that this burden of additional oversight will cause useful tests to become unavailable to patient care and new, valuable tests to cease to be developed due to the time and cost of FDA review.

Other regulatory agencies are responding to the apparent quality issues seen for molecular testing. The Secretary’s Advisory Committee on Genetics, Health and Society (SACGHS) recommends increased regulation of molecular testing and has recently published a report recommending increased genetics education and training for health care professionals.⁸ The Clinical Laboratory Improvement Amendments Committee (CLIAC) scheduled the topic of updating molecular proficiency testing regulations on its September 2010 agenda. CDC published quality assurance recommendations for molecular testing in its Morbidity and Mortality Weekly report, “Good Laboratory practices for Molecular Genetic Testing for Heritable diseases and Conditions.” Additional recommendations will be released this year. 

Although more data is needed to better define error rates for molecular testing, there is sufficient evidence to confirm that molecular tests are complex and require stringent monitoring. As James O. Westgard, cofounder and principal of Westgard QC Inc. and author of several books on laboratory quality management points out, error rates need to be known or estimated as part of the process for establishing a quality control plan. Error rates are the basis of determining quality goals and measuring success. Most laboratories are adept at implementing validation and QC plans according to CLIA for high-complexity tests in other laboratory disciplines, but QC of molecular tests involves challenges such as an incomplete understanding of how to apply traditional QC practices to molecular methods, the difficulty of implementing QC plans due to lack of software to perform the analysis, scarcity of quality controls, and lack of standardization of test methods with few FDA-approved test kits.

More resources to meet these challenges are emerging, including pending CLSI guidance MM20 “Quality Management for Molecular Genetic Testing” and updated MM01 “Molecular Diagnostic Methods for Genetic Diseases.” Recently released Basic QC Practices, Third Edition by James O. Westgard contains a discussion of the application of quality control techniques to molecular diagnostics, while the AHRQ report lists 34 documents with guidelines and standards for performing molecular tests—16 of which contain specific sections on QC and quality assurance.^9,1 Help is also available from industry colleagues and thought leaders through the listservs of the Association for Molecular Pathology (AMP) and the American Association for Clinical Chemistry (AACC) molecular division, and manufacturers are striving to provide guidance on sources for quality control materials.

Among larger molecular labs, innovation is taking place to affect efficient monitoring of total test performance through implementation of traditional QC plans designed to prevent errors and ensure accurate results.¹⁰ Such innovation includes using traditional quality control techniques developed for the chemistry laboratory, including plotting data on Levey-Jennings charts and using Westgard Rules to determine when action is required.⁹ 

Principles for the use of traditional QC practices for viral load assays and minimal residual disease is described in Westgard Basic QC Practices, Chapter 19, and posters have been presented at professional meetings describing the application of Westgard Multi-Rules to tumor-marker and minimal-residual-disease assays. A poster from the University of Pittsburgh describes programming to capture data from a viral load assay with calculations and display of QC data by an Excel program, and one molecular laboratory reported working with a vendor to develop automated QC analysis software for several in-house molecular assays.^11,12 

A well-known reference laboratory sees the value of tracking data for qualitative molecular tests to identify shifts or trends in the test system and reported on the development of a clever in-house Excel program to extract numerical output to monitor its cystic fibrosis assay.¹³ However, small to medium laboratories with limited resources are not able to dedicate the time to set up spreadsheets and enter data, and the lack of software to handle data input and analysis is a major roadblock to implementing a strong QC program for molecular tests.

Quality controls for molecular tests remain scarce, and very few are manufactured under GMP. The Genetic Testing Reference Materials Coordination Program (GeT-RM), under the auspices of CDC, arranges to have DNA from relevant Coriell repository cell lines tested in multiple volunteer laboratories to verify their performance. GeT-RM also catalogs other sources of control and reference materials; the complete list is available through GeT-RM on the CDC Web site. The AHRQ report also provides a listing of available quality controls and standards for molecular tests.¹ Yet while GeneTests lists molecular tests for more than 2000 diseases, the GeT-RM site only lists reference materials and quality controls for a few high-volume tests.⁶ Clearly more controls materials are needed, but even if current regulations create a better market by encouraging manufacturers to enter the controls business, controls for the lower-volume tests will remain a significant challenge. 

The lack of test-method standardization is a more significant problem in molecular testing than other laboratory disciplines because there are no established, traceable methods for measuring nucleic acids. Standardization is essential to ensuring that molecular test results will be the same, laboratory to laboratory and method to method. Recognizing this problem as having a significant impact on patient care, FDA requested that AACC establish a process by which tests lacking established measurement methods could be harmonized so that clinical results will be similar in all available clinical assays. A conference hosted by AACC at the National Institute of Standards and Technology (NIST) in Gaithersburg, Maryland, called “Improving Clinical Laboratory Testing through Harmonization: An International Forum,” was scheduled for the end of October 2010. NIST and the Institute of Reference Materials and Measurements (IRMM) are committed to generating standard reference materials and traceable methods for selected molecular tests, and the World Health Organization and NIST will soon release standards for cytomegalovirus (CMW) testing.¹⁴

The development of standards and harmonization of methods are slow processes, but, even without available standards, FDA-approved kits would improve quality of molecular testing by providing the labs with complete test systems manufactured under GMP and scrutinized for reproducible performance. However, FDA review is also slow and expensive, presenting a particular problem for molecular tests, most of which are low-volume tests. FDA does recognize the need to streamline the review process without sacrificing quality, and on August 4, 1010, it issued two comprehensive evaluations containing recommendations addressing three key objectives: foster device innovation, create a more predictable regulatory environment, and enhance device safety.^15-17 

A quality problem exists for molecular diagnostics across the current spectrum of both FDA-approved tests and LDTs. Progress is being made, but many challenges to the quality assurance of molecular testing remain. The typical laboratory infrastructure of traditional QC practices, software tools, control materials, standards, standardization, and FDA-approved or -cleared kits is lacking for molecular testing. CDC’s efforts to increase molecular proficiency testing will be helpful to refining the error rate, and its recommendations will continue to improve overall quality. There is increased implementation of traditional QC practice for molecular testing, but additional education and software for QC analysis is needed for widespread adoption. The availability of quality controls and reference materials continues to build, but the advent of new tests continues to outpace those efforts. More stringently applied regulations may encourage further development, but government funding may be needed to speed up the process and provide controls for rare diseases. FDA test kits alone will not solve the problem but may provide a foundation on which to build reliable quality assurance programs.

Inaccurate molecular test results can have particularly serious repercussions for patients and their families; significant effort and expense to ensure quality are justified. Improvement of accuracy and reliability must continue to be the goal of both the regulators and the industry in order to produce high-quality molecular testing for the best possible patient care. 

1. F Sun, BW, S Uhl, R Ballard, K Tipton, K Schoelles, Quality, Regulation and Clinical Utility of Laboratory-developed Molecular Tests 2010, Agency for Healthcare Research and Quality (AHRQ): Rockville, Md; Available from: www.cms.gov/determinationprocess/downloads/id72TA.pdf

2. J Marki-Zay et al., “European External Quality Control Study on the Competence of Laboratories to Recognize Rare Sequence Variants Resulting in Unusual Genotyping Results.” Clinical Chemistry, 2009. 55(4): p. 739-747.

3. S Berwouts et al., “Evaluation and use of a synthetic quality-control material, included in the European external quality assessment scheme for cystic fibrosis.” Human Mutation, 2008. 29(8): p. 1063-1070.

4. J Camajova et al., “Variability in the use of CE-marked assays for in vitro diagnostics of CFTR gene mutations in European genetic testing laboratories.” European Journal of Human Genetics, 2009. 17(4): p. 537-40.

5. R Michael. “Georgetown University Hospital Suspends Testing at One of Its Pathology Laboratories,” 2010 [cited August 19, 2010]. Available from: www.darkdaily.com/georgetown-university-hospital-suspends-testing-at-one-of-its-pathology-laboratories-809

6. NCBI. GeneTests database [cited August 19, 2010]. Available from: www.ncbi.nlm.nih.gov/sites/GeneTests/?db=GeneTests

7. July 2010 Public Meeting on Oversight of Laboratory-Developed Tests, led by FDA. Meeting webcast available from www.fda.gov/MedicalDevices/NewsEvents/WorkshopsConferences/ucm212830.htm

8. Secretary’s Advisory Committee on Genetics, Health, and Society (SACGHS), “U.S. system of oversight of genetic testing: a response to the charge of the Secretary of Health and Human Services.” (Washington, DC: HHS, April 2008), p. 276. Available from http://oba.od.nih.gov/oba/SACGHS/reports/SACGHS_oversight_report.pdf.

9. JO Westgard, Basic QC Practices, 3rd ed. (Madison, WI: Westgard QC, Inc., 2010), p. 362.

10. JO Westgard, Basic Planning for Quality, 1st ed. (Madison, WI: Westgard QC, Inc., 2000), 272.

11. RR Gullapalli, AB Carter, and KJA, “Automated Data Analysis of Real-Time PCR Data” (a poster). Presented at the Association for Molecular Pathology Annual Meeting, 2008. Published in the Journal of Molecular Diagnostics, p. 613.

12. Dartmouth-Hitchcock, personal communication from molecular diagnostics laboratory to J. Gordon, Editor. 2010. [Please explain: Is it a letter to the editor of a journal? Or something else? Need to know so I can format it properly. Thanks.]

13. MC Giese and WE Highsmith, Laboratory-Developed Tests: Validation Strategies to Meet Regulatory Requirements, J. Gordon, Editor. AACC National Meeting and Expo, Anaheim, CA, 2010.

14. M Holden et al., “Development of a NIST Standard Reference Material for Cytomegalovirus,” presented at the Association for Molecular Pathology Annual Meeting, 2009. Published in the Journal of Molecular Pathology, p. 637-638.

15. JA Shuren, Message from the Center Director, 2010 [cited August 25, 2010]. Available from www.fda.gov/downloads/AboutFDA/CentersOffices/CDRH/CDRHReports/UCM220782.pdf.

16. 510(k) Working Group. CDRH Preliminary Internal Evaluations, Vol. 1, 2010 [cited August 25, 2010]. Available from www.fda.gov/downloads/AboutFDA/CentersOffices/CDRH/CDRHReports/UCM220784.pdf.

17. Task Force on the Utilization of Science in Regulatory Decision-Making. CDRH Preliminary Internal Evaluations, Vol. 2, 2010 [cited August 25, 2010]. Available from www.fda.gov/downloads/AboutFDA/CentersOffices/CDRH/CDRHReports/UCM220783.pdf. 

 

Clark A. Rundell, PhD, is vice president for research at Maine Molecular Quality Controls Inc. (Scarborough, ME). He can be reached at rundell@mmqci.com.