Regulations & Standards

Published: January 1, 2001
Find more content on:
Calibration traceability: NIST workshop takes a major step forward

By: Donald M. Powers

Among IVD manufacturers, metrological traceability is commonly the province of the company's metrology department–a group of people that regularly come around to calibrate spectrophotometers, pH meters, and other measuring devices according to standards approved by the National Institute of Standards and Technology (NIST). The traceability of calibrators used in the performance of IVD assays–in the metrological sense–is a concept promoted by the European Union's IVD Directive.1,2

Recognizing the impact that requiring traceability will have on U.S. IVD manufacturers, NIST hosted a workshop in November 2000 on measurement traceability for the clinical laboratory. The workshop was intended to increase awareness of issues related to metrological traceability and to define the role that NIST will play in ensuring that calibrators are traceable to recognized true values.

Many of the invited speakers were from key European organizations that are intimately involved in developing aspects of the IVD Directive. Together, their presentations defined the infrastructure of a reference system that would include certified reference materials, approved reference methods, and a network of accredited reference laboratories.

Essential Elements

The speakers were less clear about how such a system could best be established in the face of uncertain requirements, international politics, and the competing priorities of an industry under heavy cost constraints. To create such a reference system, organizers will have to determine how it will be funded and who will do the work; identify appropriate certifiers, approvers, and accreditors; and decide what the system's priorities will be. In five breakout sessions, the workshop participants brainstormed solutions to the problem of traceability and converged on the following essential elements for a reference system.

  • The infrastructure of the reference system must be global and should make use of existing organizations and initiatives. National metrology institutes (NMIs) should enter into mutual recognition arrangements (MRAs). Participating organizations will have to find ways to transcend politics.

  • A network of reference-material producers and reference laboratories is needed to support the assignment of specific values to calibrators. Such suppliers and laboratories must be capable of operating in an FDA-regulated environment, must conform to the new ISO standards for reference materials and laboratories, and must be sensitive to manufacturers' need for rapid turnaround.

  • The IVD Directive does not mandate the development of new reference materials or methods. Where NMIs have programs that are intended to develop such materials or methods, they should be prioritized and developed according to the needs of the medical community.

  • Interpretive guidance for industry would promote consistent interpretation of regulations and standards relating to calibrator traceability.3–5 To help small manufacturers meet such requirements, there should be improved communication about available resources (e.g., via a central Web site).

  • There is significant potential for introducing medical errors if changes in laboratory practices are introduced in an uncoordinated manner. To minimize adverse effects on laboratories, physicians, and patients, there will need to be a transition plan for any calibration changes that must take place in connection with the implementation of traceability to reference materials or reference methods.

  • Application of calibrator traceability is different from demonstrating accuracy relative to intended use. The benefits of applying traceable calibrators should be evaluated on the basis of medical outcomes. The cost-effectiveness of the newly mandated approaches to establishing traceability should be demonstrated.

Among the workshop attendees there was general consensus on these overarching recommendations. However, few specific solutions beyond these general principles were proposed.

International Consortium

At the workshop, U.S. participants seemed surprised to learn about the initiatives already under way in Europe. The Netherlands Metrology Institute has already organized a broad-based consortium of major stakeholders to define and plan the infrastructure necessary to support IVD Directive requirements. The European Commission is already considering a funding request to support this ambitious program.

The formative meeting of the consortium was attended by NMIs from Denmark, Germany, The Netherlands and the United Kingdom, as well as representatives of the International Federation of Clinical Chemistry and Laboratory Medicine (IFCC), the European Commission's Institute for Reference Materials and Measurements (EC-IRMM), the German Society for Clinical Chemistry (DGKC), and the European Diagnostics Manufacturers Association (EDMA). But the Europeans are also thinking globally: also in attendance were senior representatives of NIST and the Australian NMI, both of which were invited to join the consortium.

Notably absent from the list of attendees at the consortium meeting were representatives of AdvaMed, the American Association for Clinical Chemistry (AACC), the College of American Pathologists (CAP), the National Committee for Clinical Laboratory Standards (NCCLS), the World Association of Pathologists (WASP), the World Health Organization (WHO), and any of the counterpart organizations from Asia and Latin America. (In September 2000, WHO held its own summit conference to define its role in supporting calibration traceability. That conference was attended by many of the organizations that attended the consortium formation meeting.)

It is not obvious which organization might emerge as the natural leader of the consortium. Attendees generally agreed that the lead organization could not be a national institute or society and that it should have international recognition. That leaves only the following few candidates.

International Federation of Clinical Chemistry and Laboratory Medicine. IFCC is generally acknowledged as the international authority on scientific matters related to clinical laboratory testing. At the NIST workshop, one breakout group suggested that IFCC should take responsibility for the reference system infrastructure, but another group questioned IFCC's ability to handle the logistics of such a major undertaking because of its lack of full-time executive management and sufficient administrative staff. Furthermore, despite the fact that AACC is one of the 70 professional societies that make up IFCC, U.S. clinical chemists generally regard IFCC as a competing European organization. Industry's role in IFCC has increased over the past few years–more than 40 IVD manufacturers are now affiliated with IFCC, and one nonvoting seat on the IFCC board is reserved for an industry representative.

National Committee for Clinical Laboratory Standards. Despite recent attempts to be recognized for its global activities such as administering the ISO technical committee for clinical laboratory testing (TC212), NCCLS is still perceived by Europeans as a U.S. organization. NCCLS has the experience and staff to manage a reference system infrastructure, but may not have the interest to do so. NCCLS already possesses the framework for such a reference system in the form of the highly regarded National Reference System for the Clinical Laboratory (NRSCL), which could be reborn internationally as the IRSCL.

World Health Organization. WHO is the brand-name leader for reference materials, and WHO sponsorship would bring instant credibility to an international reference system. However, WHO operates programs only through collaborating centers that organize and manage programs on its behalf. Most WHO standards do not currently meet ISO requirements for primary reference materials, but WHO could address that incongruity by adopting the new ISO standards for future reference materials. WHO sponsorship of the consortium remains a reasonable option.

At the NIST workshop, no other international organizations emerged as candidates to lead the consortium. One concept for involving all the candidates would be for the leadership of the consortium to rotate among them periodically, with a stable secretariat organization being appointed to manage day-to-day operations. Another concept would build on the strengths of all three organizations, creating a transcontinental partnership with WHO as sponsor, IFCC providing scientific direction, and NCCLS managing the program. Directors would be elected by the worldwide stakeholders, who would collectively fund the consortium.

Communication and Industry Guidance

For many industry attendees at the NIST workshop, the traceability requirement was news. Some company representatives complained about lack of information regarding the requirement, even though discussions about the IVD Directive and the new European standards have been going on for more than 6 years. Industry has free access to information about available reference materials via the Web sites of NIST (http://www.nist.gov), IRMM (http://www.irmm.jrc.be), and WHO (http://www.who.ch).

Nevertheless, attendees agreed that it would be advantageous to develop a guidance document that would promote consistent interpretation of the directive and supporting standards. To ensure that the guidance addresses industry issues and is written in language that industry understands, attendees agreed that the responsibility for writing industry guidance should belong to trade associations.

Reference Material Gaps

Only 25–30 clinical analytes currently have certified primary reference materials, and even fewer analytes have recognized reference methods. The agendas of both the NIST workshop and the consortium formation meeting included discussions of ways to close gaps in available reference materials–that is, to develop more of them.

It has not escaped the attention of NMIs that the traceability requirements of the IVD Directive open up a significant marketing opportunity for new reference materials. While expanding the list is not inconsistent with the intent of the IVD Directive, traceability is required only when an "available reference material" or "available reference method" exists.

Manufacturers are not anxious for the reference system to include more analytes at this time. It makes sense to implement traceability first for those analytes that have reference materials and reference methods. Then, once a workable infrastructure is in place, the system should be expanded to cover other analytes developed according to medical priorities. To ensure that this industry view is taken into account when decisions are shaping the future of traceability, it is important that manufacturers participate actively in the discussions.

Reference Laboratory Network

Manufacturers of cholesterol assays know the value of having an external reference laboratory available to resolve disputes over accuracy. As a service to manufacturers and clinical laboratories, the Centers for Disease Control and Prevention (CDC) organized and coordinated a network of cholesterol reference laboratories for the National Cholesterol Education Program, which resulted in a dramatic improvement in the accuracy of cholesterol measurements in the United States, and ultimately in the rest of the world. However, this improvement was accomplished without reliance on metrological traceability. Most manufacturers used the network to analyze splits of patients' specimens in order to verify the effectiveness of their process for assigning calibrator values–not to actually assign those calibrator values.

What is envisioned now goes well beyond that simple challenge. Requirements for maintaining a reference laboratory using true reference methods are so demanding and costly that many manufacturers might be willing to outsource their entire process for assigning calibrator and control values to an independent, accredited reference laboratory. In fact, small manufacturers may have little choice but to do so. Even so, it is questionable whether there is sufficient need to warrant establishing a network of such reference laboratories, whether existing labs are qualified to provide such services, or whether manufacturers are willing to pay for them.

At the NIST workshop, some attendees were from university and hospital laboratories interested in evaluating the business case. If it turns out that there is sufficient need, then an international system must be put in place to accredit reference laboratories according to the new ISO 15195 standard.6

Transition Period

Little has been said about how calibration changes driven by the IVD Directive will be implemented. Every manufacturer and clinical laboratory knows that change in the calibration of an assay involves disruption at several levels. Indeed, many laboratories adjust the manufacturer's calibration to produce results more consistent with their historical reference intervals. As manufacturers rush to CE mark their existing products by 2003, the transition to traceable calibrators must be managed carefully to avoid errors that could affect patient care.

A transition plan is needed to minimize adverse effects on laboratories, physicians, and patients. The professional societies should take the lead on this, with assistance from industry trade associations, individual companies, and government regulatory agencies.

Another transition issue that requires more debate and guidance from regulatory authorities is how much time manufacturers will have to implement a calibration correction once a new reference material becomes available, and how this change should affect products already on the market. When a manufacturer implements a calibration change, it is typically applied to all product in the field rather than allowing two different sets of values to be reported. Will regulatory authorities consider these field corrections to be recalls?

Cost-Benefit Evaluation

At the end of the initial phase–before larger sums are spent developing metrological reference systems for the several hundred remaining analytes–the professional societies should collectively evaluate whether the traceability requirement is worth the cost. At the NIST workshop, several speakers cited a German study that claimed that 30% of laboratory values had to be repeated due to lack of traceability. The real figure is almost certainly much lower, but it is not zero. Other speakers described clinical studies and anecdotes that made a more compelling case for comparability from lab to lab–at least for the analytes they discussed.

It is difficult to argue that accuracy and comparability are not important in a mobile society. But, keeping the cholesterol success story in mind, metrological traceability may not be required for all analytes. Approaches to establishing traceability should be evaluated based on medical outcomes to verify that the benefits outweigh the costs.

Next Steps

Workshop participants identified several necessary actions–but in most cases they were not able to specify who should take them, or when. The underlying assumption seemed to be that the European-organized consortium will assign responsibilities to its members. The assemblage asked for another meeting in 6 months to continue the dialog and review progress, but this request is questionable, considering the urgency imposed by the IVD Directive deadline. Now is the time for action, not more debate.

The NIST workshop raised a number of important issues and proposed a number of solutions. However, most of the participants were stakeholder representatives who attended primarily to acquire more information; they were not decision makers who could commit resources to an action plan.

The next step should be a smaller meeting to create a plan. Invitees should be limited to the leadership or empowered delegates of the key organizations already identified–in essence, the consortium plus the missing stakeholders. That meeting should define the group's mission and operating charter, select leadership, and appoint a representative steering committee of a workable size. This group ought to move forward and keep us all informed of progress through the member organizations.

References

1. "Council Directive 98/79/EC of the European Parliament and of the Council of 27 October 1998 on In Vitro Diagnostic Medical Devices," Official Journal of the European Communities L331 (7 December 1998). (Also available on the Internet at http://www.edma-ivd.be).

2. Donald M Powers, "Traceability of Assay Calibrators: The EU's IVD Directive Raises the Bar," IVD Technology 6, no. 4 (2000): 26–33. (Also available on the Internet at http://www.devicelink.com/ivdt/archive/00/07/003.html).

3. In Vitro Diagnostic Medical Devices–Measurement of Quantities in Samples of Biological Origin–Metrological Traceability of Values Assigned to Calibrators and Control Materials, ISO/prEN 17511 (Geneva: International Organization for Standardization, 1999).

4. In Vitro Diagnostics Systems–Measurement of Quantities in Samples of Biological Origin–Preparation of Reference Measurement Procedures, ISO DIS 15193 (formerly EN 12286: 1998) (Geneva: International Organization for Standardization, 1999).

5. In Vitro Diagnostic Medical Devices–Measurement of Quantities in Samples of Biological Origin–Description of Reference Materials, ISO DIS 15194 (formerly EN 12287: 1998) (Geneva: International Organization for Standardization, 1999).

6. In Vitro Diagnostic Medical Devices–Measurement of Quantities in Samples of Biological Origin–Requirements for Reference Measurement Laboratories in Laboratory Medicine, ISO DIS 15195 (Geneva: International Organization for Standardization, 1999).

Donald M. Powers, PhD, is president of Powers Consulting Services (Rochester, NY), an independent IVD regulatory and quality consulting firm.

 

 

 

 


Copyright ©2001 IVD Technology


No votes yet