Sneak Preview: What Will the Quality Metrics Initiative Look Like?

I had lunch in a restaurant the other day. During my meal, I noticed that there weren't many other customers. I didn't give it much thought until I walked out and read the restaurant's hygiene grade (posted in its window for everyone to observe) was a "C". Despite the frustration that I should have noticed the grade before I entered (d'oh!), not to mention concerns for my health (was that a roach burger with cheese I just woofed down?) my experience prompted some thoughts on FDA's emerging Quality Metrics initiative.

There's been much discussion regarding FDA's pursuit of a Quality Metrics regulation for industry. Regulatory agencies and certain pharmaceutical industry groups (ISPE and PDA, for example), have suggested that quality differences between products and firms should be made more visible to the public. The use of scorecards or rankings based on quality metrics has been an effective tool for other industries. An example of this is issuance of restaurant hygiene grade cards.

So, how is this program going to look and what is its impact above and beyond what many companies are already doing with respect to their quality metrics activities? I recently participated in a University of Southern California poll seeking industry input regarding quality metrics programs for drug companies. A review of the questions and options for answering them provides insight as to the current regulatory thinking and where they're likely heading with the impending requirement. To be sure, it is clear that FDA is shifting their focus to quality system performance and away from compliance (as quality system compliance is already a legal requirement and therefore a "given" in the equation).

The following questions are extracted from the poll. See what you can deduce based on the nature of the questions, the language used, and the options for answers. One thing is clear to me: With the FDA's evolving approach to quality, how a company collects and uses metrics will also identify a company's risk for quality failures -- and whether or not those risks warrant added scrutiny and inspections by the FDA. Good performance data = less regulatory inspections with the opposite being true.

Data trending and systems analysis can provide valuable insight into quality performance. Which type of data outputs listed below best describes the types of analysis you or your organization commonly use?

• Non-Compliance Trend Reports
• Correlation Between 483s and Warning Letters
• Number of Recalls or Market Withdrawals
• Product Defect Rates
• Customer Complaint Rates
• Number of Corrective Actions Taken
• Adverse Event Rates

Which of the following, if any, might your company be willing to share in an effort to help improve industry-wide quality metrics?

• Complaint Rates
• Manufacturing Error Rates and Trend Analysis
• Adverse Event Rates
• Products Return Rates
• Non-Conformance Rates
• Number of Corrective Actions (CAPA)
• Product Rejection Rates
• Raw Material Supplier Metrics
• Process Performance Indicators (e.g.CpK)

Thinking about your customers and buyers, what quality metrics do you believe will be most useful to help them determine the overall quality and supply reliability of your product?

• Complaint Rates
• Recall Rates
• Number of FDA Actions (483s, Warning Letters)
• Health Agency Inspection Track-Record (e.g. FDA)
• Manufacturing Error and Failure Rates
• Emergency Inventory Held
• Number of shortages experienced
• Number of Back-up Manufacturing Sites
• Measure of Product Robustness
• Adverse Event Rates

Please rank in order of preference what type of organization is best suited to evaluate quality performance and issue a quality rating.

• Regulatory Agency (e.g. FDA)
• Independent Government Agency
• Private Organization (e.g. JD Edwards)
• Industry Trade Group
• Other Non-Profit Organizations
• Consumer Groups

In your opinion, what would be the best way to reflect a quality rating for FDA-regulated products?

• Alphabetical Grade (A,B,C)
• Definitive Numerical Value (1-100%)
• Graded Numerical Value (like a credit rating)
• Color Card (Green, Yellow, Red)
• Numerical Ranking Among Industry (like hospital and college rankings)

In your opinion, what amount of information would a consumer need to know regarding a quality ranking?

• Quality Grade
• Description of Key Metrics Associated with the Grade
• Grade Determined for Each Key Metric
• Description on Why a Lower Grade Was Issued
• Date of When Grade was issued

If quality grades were issued to firms, do you believe that it should be mandatory to disclose them publicly?

• Yes
• No

There are other poll questions not included here but they are similar in content, basically exploring the type of metrics to be used, how to set-up a rating standard, and how to best divulge the information publically or otherwise.  Think about it, the possibility that a pharmaceutical company's quality rating, essentially equivalent to a restaurant's "hygiene rating", will be in the public domain has some very interesting business ramifications.  There will certainly be companies that will take a hard look at the effectiveness of their quality systems if they're assigned a poor grade.

One can also assume that FDA will be changing their audit approaches to include greater emphasis on quality system performance.  I'll write more about that in a future blog.