Ironbarque

Medical Device Consulting Regulatory Quality Complaince

Risk of the Device as a Therapy

For any given disease state, there is the option to not treat, or to treat selecting an appropriate treatment option. There are often more than one forms of treatment possible. In essence, risk management is about establishing that a device has a positive risk-benefit ratio when compared to no treatment, other treatments and similar devices.  This positive risk-benefit ratio has to be maintained over the product life or corrective action taken.

What the Regulations Require

Different Laws but a Common Objective

The medical device world is loosely divided into the US Market under FDA Regulations and the majority of other countries where reference to ISO standards and regulations are typical of those seen in the EU.  The discussion here is limited to these two markets, however Japan, Australia, Canada and a host of other countries have regulatory frameworks not dissimilar to that seen in the EU. This is no accident but the result of an active process that started in 1992 with the Global Harmonization Task Force and now continues under the International Medical Device Regulators Forum

Safe & Effective

Misbranding

Medical Devices are regulated under the Federal Food Drug & Cosmetic Act. The language of the Act and of the regulations promulgated by the FDA under the Act makes frequent reference to the words "safe" and "effective". Most of the authority to remove devices that are not safe and effective relies on the FDA showing that a device is adulterated (not what is should be) or misbranded (not labeled correctly). Within the requirements for labeling in the Act, are the requirements that the instructions for use are adequate in instruction and warnings; and that the device is not health endangering when used as instructed. To see this section of the FFDCA, click

As clinical practice changes and improves, what constitutes appropriate labeling changes over time as medical practice evolves. New warnings are required. Instructions for Use have to adapt to changes in clinical practice. The standard for health endangering changes with improvements in treatment modalities and treatment outcomes.

As an example, pacemakers with anti-tachycardia pacing were initially well received until it became obvious that this feature had to be coupled with the ability to cardiovert induced ventricular fibrillation i.e. this feature had to be part of an automatic implanted cardioverter -defibrillator to be safe.

510k Filings are not Static

It used to be that a 510k was filed against a product and then nothing was done to maintain the design history file unless a major change to the product was required. The product was the product was the product was the product.  

Once the FDA required the use of consensus standards as a form of "special control"; 510k filings were brutally shaken from their dormancy.  The consensus standards require the maintenance of the design in terms of its risk-benefit ratio in a way that is not explicit in 21 CFR 820. This regulation speaks to risk analysis where appropriate - that casual term that actually means "do it unless you can prove that you do not need to do it".  The consensus standards now say where it is appropriate.

The FDA has revised its regulations and guidance on 510k devices, and is re-enforcing these changes in its audit program, to require reassessment of risk with each change and to require the feedback of data from medical device reporting/ the complaint file into the risk management file for the device. A review of the Warning Letters for 2014 and 2015 will show this enforcement effort leading in a few cases to injunction and consent decree.

ISO Countries: Risk-Benefit

The EU is more explicit about its requirements. There are three main directives:

Using the MDD as an example, the first section of Annex I to the Medical Device Directive is explicit about the need to demonstrate that "Any side effects or undesirable conditions must constitute acceptable risks when weighed against the performances intended." For the text - click . There is also the requirement that "Demonstration of conformity with the essential requirements must include a clinical evaluation in accordance with Annex X."  Annex X explicitly requires that the manufacturer "close the loop" by actively updating this clinical evaluation based upon ongoing monitoring of the postmarket performance of the device. For the text of the General Provisions - click .

Australia, Japan, Canada and other countries, that focus on using ISO Standards as a support to their regulations, employ regulations with similar requirements.

Medical Judgement

Risk assessments are best when based upon well documented literature reviews conducted with strong, and unbiased, medical reviewers. The Engineering team is driven to time and budget. It is a strong team that can remain impartial and dispassionate when assessing the meaning of clinical journal articles when this understanding may negatively impact bonus or career objectives.  Good people are good people; supporting them with good systems makes their job easier.

It is also important to make the review process formal.  Notes taken from quick hallway conversations with customers do not constitute a medical review.  This has to be a formal and thorough review.

Using customer physicians or nurses is often very useful however, the formality of the process is important. A question asked as "How concerned are you if "this failure" occurred?" will often elicit a response of "Not that concerned".  However this answer is given on the thought that "well, I see this often enough that I am not stressed too much and know what to do to fix things".  The correct question is "what happens if "this failure" occurs and you are not there to help the patient?"   Your answer may be surprisingly different if you ask the question in this way. The next good follow-on questions is: "And how mad are you at the Company responsible when it happens?".

Risk Literacy

One of the issues facing Medical Device firms when establishing their Clinical Evaluation (CER) documentation and in writing Health Hazard Evaluations (HHE) is properly being able to understand and present the risks involved. Are you risk literate - try this quick quiz.

The staff generating the CER or HHE need to be not only risk literate but also health literate i.e. they need to know not only the statistical basis of risk assessment but also how to frame the data being reviewed in the context of the current understanding of clinical "best practice" so that the ultimate customer, the patient, can be properly informed.  This information often comes via a learned intermediary such as a physician or a regulated process such as a Field Safety Notice or Class I or Class II recall. So the writing has to be pitched according to the means by which the data will be finally transferred. The US Center for Disease Control and Prevention (CDC) has an excellent publication on Health Literacy and how to Develop Materials to support health literacy.

The Process Elements

Design Input

Assuming that we are talking about a new device, the design inputs have to be established. What is this device to do, in which patients and to what expected outcome?  This is the start of the Clinical Evaluation of the product and a corner stone document that must be maintained as postmarket experience is garnered.

Harm/Hazard Assessment

ISO 14971 and ISO 62366 both require the identification of harms/hazards as a process requirement.  The most recent revisions to design standards such as ISO 10993 (and subparts) and ISO 60601 (and subparts) also require this activity in a design specific sense.

It is important not to limit the hazard analysis. If the device requires the use of an anesthetic, local or general, for its implantation or use, then the harms associated with this anesthetic must also be considered. The means of implanting the device and the surgical risks of this implantation must be considered even where the harm does not imply device failure.  Informing the User of the device of "expected" complications is required by device regulations.

Stratify the harms as you would with an RPN scale and match these levels to regulatory reporting levels:

Level

Harm

USA

EU/ISO

1

Death or Serious Injury

MDR Report / Class I Recall

Vigilance Report / FSCA

2

Harm - not serious

Class II Recall

FSCA Possible

3

No Harm but some form of noncompliance

Class I recall

Postmarket surveillance / Continuous improvement

4

No Harm - No Delay -
Device Cosmetic impact only

No recall

No Action

Using this approach ensures that when the unspeakable happens - you are prepared. Your documentation reflects the harm and you know, in advance, the level of response required.

Failure Analysis

Harm and failure analysis go hand in hand. Often, there is a failure of the design or in manufacture or in the use of the device behind the harm. For each failure, it is efficient to consider the labeling requirements associated with the failure and to warn of the harm associated as well as providing mitigating instructions. Instructions alone do not obviate the need to design around a failure mode nor do they make the failure mode "acceptable". The overall risk-benefit analysis must include the cost of such failure modes.

Validations

Validation sample sizes and extent drive the cost and duration of a product development project. To legitimately minimize these use normal statistics where ever possible.

Validations: Stay Normal

Where ever possible find outcome measures that are normally distributed. "Six sigma" techniques greatly simplify many validation activities. Remember that:

  1. Variations lot to lot with raw materials such as polymers can significant exceed those seen within a set of process runs on a single lot of material.

  2. Do not simply use the measured upper and lower confidence limits obtained in early design verification testing as the specification limits for design validation testing. Point 1 applies!

  3. Design validation should be undertaken on initial production units; lots, runs or equivalents. Understand your process variability before attempting design validation.

  4. Understand the error in standard deviation (SD) - see the graph below. Although you can estimate a SD from two samples, the error in the estimate is quite large. You are taking a risk, with small sample sizes, that your outcome is wrong. Maybe it is wrong in a good way - you overestimate your SD and still pass. Maybe it goes the other way .

Derived from: Ahn S and Jeffrey A. Fessler JA1.

There is no absolute standard for how large a sample size should be chosen. A simplified table, below, shows the percent error in the standard deviation as a function of sample size. As an example, if you are using +/-3SD equations; with a sample size of 7 you could actually be somewhere between 2 and 4 SD (i.3. 3 x 30% ~ 1 SD).

Sample Size

% Error

7

30%

14

20%

22

15%

47

10%

170

5%

Validations: If you Must be Binary

The advisory annexes to ISO 14971 speak to the stratification of risk as the basis for determining the acceptability of a failure mode and effectively the sample size required in a validation to prove that the failure does not occur. There is an inbuilt flaw in this logic. The more novel the device or for a new feature to an existing device, there is little history available against which to rate the probability of a failure. This is where risk illiteracy and "gaming" come into play.

The common flaw is to permit the reduction of sample size requirements for an hazardous failure mode because it is rare. This is particularly done when the test required has a binary outcome and c=0 type testing strategies are used - see the table below.

0 Failures in
At 95% Confidence Excludes a Failure Rate Greater than
10
30%
30
10%
100
3%
`300
1%
1000
0.3%
30000
0.1%

Everyone understands that large failure rates are not acceptable.  We take care of the 10% problems. 10% problems are not that costly to find via test. To find a 10% issue, at 95% confidence, test 30.

When failure modes are truly uncommon, say 0.01%, one is often safe in minimizing the "risk" driven work for the failure mode even if the harm is significant. For a binary outcome test, the 95% confidence sample size to exclude a 0.01% failure is 30,000 units. Have you actually ever seen someone do this? Typically, these situations are avoided by finding an outcome that is normally distributed and using the power of "sigma" or by writing the procedural set such that a lesser level of confidence is required by fiat for uncommon failure modes.

Companies can do this and not hit a major problem because, for many devices, the sales per annum are modest.  It takes a long time to see one failure and, provided that this failure is recognized in the Instructions for Use / Design History File in an appropriate manner; this failure is treated as an isolated incident.

Pity the company that sells 2 million units a year, however.  A new systematic failure that causes death at a rate of 0.01% in a device that sells at the rate of 2 million per annum will have to be managed as a recall. 200 unexpected deaths per annum is probably not acceptable to most regulatory bodies.

For failure modes where the rates run in single digit to large fractions of percent; the tendency is to minimize the rate and hope. A 1% failure rate is downgraded to a 0.1% and proclaimed not to be significant.  Frequently, their is a robust redundancy in any design.  The natural tendency of engineers to add "50% for safety" protects.

But what happens, if it is a 1% failure rate and this failure rate occurs. Now the Company is faced with a significant increase in expected failure rate.  If a very harmful failure, a recall is required. The Company is, by definition, non-complaint because it failed to properly assess the risk.  This occurs not only with binary outcome testing but also with outcomes that are normally distributed where the lot to lot variability has a significant independent effect.

Risks Analysis

Assuming that you have made the device to meet its specifications and given what you know about the complications / benefits associated with its use in properly qualified hands then how does this device compare to no treatment, other treatment modalities and other similar devices?

Evaluation vs Investigation

For US based firms, the question of whether a Clinical Evaluation or a Clinical Investigation are required in the EU can arise.  The answer is simple.  If you are doing an IDE in the USA, then you are generally doing an IDE in the EU. De novo devices that are down classified to Class II in the USA will probably require a clinical investigation in the EU.

1. Ahn S and Jeffrey A. Fessler JA. Standard Errors of Mean, Variance, and Standard Deviation Estimators. EECS Department The University of Michigan July 24, 2003. www.eecs.umich.edu/~fessler/papers/files/tr/stderr.pdf Link verified 2015-12-06