Your trusted CER partner

9. Appraisal of pertinent data (Stage 2)

9.1. General considerations

In order to determine the value of the data identified in stage 1, the evaluators should appraise each individual document in terms of its contribution to the evaluation of the clinical performance and clinical safety of the device.

Uncertainty arises from two sources: the methodological quality of the data, and the relevance of the data to the evaluation of the device in relation to the different aspects12 of its intended purpose. Both sources of uncertainty should be analysed to determine a weighting for each data set.

The evaluators should therefore:

• identify information contained in each document,
• evaluate the methodological quality of work done by the authors and from that, the scientific validity of the information,
• determine the relevance of the information to the clinical evaluation, and
• systematically weight the contribution of each data set to the clinical evaluation.

9.2. The appraisal plan

To ensure systematic and unbiased appraisal of the data, the evaluators should set up an appraisal plan that describes the procedure and the criteria to be used for the appraisal.

• The appraisal plan typically includes:
  • criteria for determining the methodological quality and the scientific validity of each data set.
  • criteria for determining the relevance to the clinical evaluation (relevance to the device and to the different aspects of its intended purpose).
  • criteria for weighting the contribution of each data set to the overall clinical evaluation.

• The appraisal should be thorough and objective, i.e. it should identify and attribute adequate weighting both to favourable and unfavourable contents of each document.
• The criteria adopted for the appraisal should reflect the nature, history and intended clinical use of the device. They should be documented and justified on the basis of current knowledge / the state of the art, applying accepted scientific standards.
• There are many acceptable ways, both qualitative and quantitative, by which the appraisal can be carried out13. For many well established devices and lower-risk devices, qualitative data may be adequate to fulfil the requirements of the MDD and AIMDD. The evaluation criteria should be adjusted accordingly.
• The appraisal plan should be documented in the clinical evaluation report.

9.3. Conduct of the appraisal

The evaluators should

• follow the pre-defined appraisal plan strictly and apply its criteria consistently throughout the appraisal;
• base their appraisal on the full text of publications and of other documents (not abstracts or summaries), so as to review all of the contents, the methodology employed, the reporting of results, the validity of conclusions drawn from the investigation or report, and evaluate any limitations and potential sources of error in the data;

• document the appraisal in the clinical evaluation report to the extent that it can be critically reviewed by others.

9.3.1. How to evaluate methodological quality and scientific validity

The evaluators should examine the methods used to generate/ collect the data and evaluate the extent to which the observed effect (performance or safety outcomes) can be considered to be due to intervention with the device or due to

• confounding influences (e.g. the natural course of the underlying medical condition /regression to the mean, concomitant treatments)
• bias
• random error
• inadequate disclosure of information
• misinterpretation

Some papers considered unsuitable for demonstration of adequate performance because of poor elements of the study design or inadequate analysis may still contain data suitable for safety analysis or vice versa.

Examples of aspects that can be taken into consideration for evaluating the methodological quality and the scientific validity of the evidence are detailed below.

1. Study design of pre-market and post-market clinical investigations
   Considerations may need to include:

• adequacy of the sample size and power calculation
• adequacy and relevance of endpoints (including validity of surrogate endpoints, if used)
• adequacy of applied controls (including choice of the study type and of comparators, if applicable)
• prospective randomisation of patients (in case of multiple treatment arms)
• adequacy of inclusion and exclusion criteria, and of stratification of patients (e.g. in respect to age, medical indication, severity of the condition, gender, other prognostic factors)
• distribution of prognostic factors (in case of multiple groups, were the groups comparable for these factors?)
• blinding of patients (may include use of sham devices or sham surgery), professional users, outcome assessors (blinded endpoints)
• adequacy of the follow-up period, including if follow-up was long enough for outcomes to occur, and if follow-up was frequent enough to detect temporary side effects and complications (such as prolonged wound healing)
• reliability of the methods used for quantifying symptoms and outcomes (including validation of the methods)
• adequate recording and reporting of serious adverse events and device deficiencies
• adequate handling of medications and concomitant interventions
• adequacy of procedures for retrieving complete information (e.g. procedures to be applied when contacts with patients are lost, disclosure of reasons for patients leaving the study, conduct of sensitivity analysis for determining if missing data affect conclusions)
  The evaluators should verify whether clinical investigations have been defined in such a way as to confirm or refute the manufacturer’s claims for the device; and whether these investigations include an adequate number of observations to guarantee the scientific validity of the conclusions.

The evaluators should verify whether clinical investigations have been defined in such a way as to confirm or refute the manufacturer’s claims for the device; and whether these investigations include an adequate number of observations to guarantee the scientific validity of the conclusions.

• Additional aspects for appraisal of the quality of clinical investigations generated and held by the manufacturer
  Where a clinical investigation has been carried out by or on behalf of a manufacturer, it is expected that documentation relating to the design, ethical and regulatory approvals, conduct, results and conclusions of the investigation needed for the clinical evaluation will be available for consideration, as appropriate. These may include:

1. the clinical investigation plan;
2. clinical investigation plan amendments and the rationale for these changes;
3. case report form templates, monitoring and audit records;
4. the relevant ethics committee documentation;
5. regulatory authority approvals as required by applicable regulations;
6. the signed and dated clinical investigation report (for investigations that are terminated);
7. the latest intermediate report available and the latest collation on serious adverse events (for investigations that are ongoing);
8. when a clinical investigation is conducted outside of the EU, an analysis whether the results are transferable to the European population;
9. a gap analysis, when a clinical investigation is conducted to standards different from EN ISO 14155; the gap analysis should contain sufficient information to be read and understood by an independent party.

The clinical investigation plan sets out how the study was intended to be conducted. It contains important information about the study design such as the selection and assignment of participants to treatment, masking (blinding of participants and investigators) and measurement of responses to treatment, which may be important sources of bias that can be assessed and possibly discounted when trying to determine the actual performance of the device. In addition the clinical investigation plan sets out the intended participant follow-up, approaches to statistical analyses and methods for recording outcomes, which may impact on the quality, completeness and validity of results obtained for performance and safety outcomes.
Also, by having the clinical investigation plan, its amendments and the clinical investigation report available, the evaluators will be able to assess the extent to which the investigation was conducted as planned and, where deviations from the original plan have occurred, the impact those deviations had on the veracity of the data generated and the conclusions that can be drawn from the investigation about the performance and safety of the device.
The clinical investigation report should be signed by the sponsor and the coordinating or principal investigator to provide assurance that the report is an accurate reflection of the conduct and results of the clinical investigation.
Another important consideration of the evaluation will be to assess whether the conduct of the investigation was in accordance with applicable regulations, and in accordance with the current applicable ethical standards that have their origin in the Declaration of Helsinki. Clinical investigations not in compliance with applicable ethical standards, medical device standards (for example EN ISO 14155 or comparable standards) or regulations should not be used for demonstration of performance and/or safety of the device. The reasons should be noted in the report.

• Information derived from vigilance data, device registry data, case series, patient dossiers, and other use data
  Evaluators need to consider significant differences between sources of information in respect to:

• procedures used for retrieving information about outcomes
• quality aspects of registers and patient dossiers

In case of information based on vigilance reporting, evaluators should consider that expected undesirable side-effects and complications of devices are not reportable under the vigilance reporting system. Under-reporting or lack of reporting of expected side effects or complications by users is common. Therefore, the vigilance system does not typically deliver adequate information about the true frequency of expected undesirable side-effects and complications. Systematic scientific data are needed for such purposes. Vigilance data, including trend analysis, should be used for identification of unexpected risks.

In case of information based on device registries, case series, retrospective analyses of patient dossiers, and other use data, the retrieval of information about outcomes may be incomplete and unreliable (have all the patients been considered? are the patients representative of the use of the device? did the register/ professional lose contact with patients if they moved on to different professionals? was there a passive or active follow-up of patients by the professionals involved? for how long?). Significant differences may exist between device registries. For instance, they may offer an important or limited coverage of a country. The evaluators should take into account the possibility of patients leaving the coverage of a registry or the follow-up of a professional when experiencing serious adverse outcomes. In routine practice, there are also significant differences in the duration of the follow-up of patients by surgeons and other professionals, and in the quality of patient dossiers and data retrieval.

For clinical experience data it is important that any reports or collations of data (e.g. the manufacturer’s PMS reports) contain sufficient information for the evaluators to be able to undertake a rational and objective evaluation of the information and make a conclusion about its significance with respect to the performance and safety of the device in question.

Reports of clinical experience that are not adequately supported by data, such as anecdotal reports or opinions, may contribute to the evaluation, e.g. for the identification of unexpected risks, but should not be used as proof of adequate clinical performance and clinical safety of the device.

• Data processing and statistics
  Aspects to consider may include:

• suitability of methods for data processing (transforming data that are suitable for analysis), converting data to a consistent format, reconstructing missing statistics from other statistics, dealing with missing data;
• exclusions from the analysis and their implications (including disclosure and adequacy of the intention-to-treat and per-protocol populations, disclosure of results from both the intention-to-treat and the per-protocol populations);
• adequacy of statistical methods.

• Quality assurance
  • compliance with Good clinical practice (GCP), such as EN ISO 14155 or equivalent standards;
  • compliance with the clinical investigation plan, independent monitoring and auditing;compliance with legal requirements.
    While a publication in a renowned peer reviewed scientific journal is generally accepted as an indicator of scientific quality, such publication is not considered an acceptable reason for bypassing or reducing appraisal activities.
• Report quality
  Evaluators should consider:

• adequacy of disclosure of methods used
• adequacy of disclosure of data, including
  • completeness of the reporting of adverse events and outcomes
  • sufficient description about the distribution of prognostic factors in the study population and in different study arms
  • disclosure of all the results the study was originally designed to generate
• validity of conclusions drawn by the authors (example: conclusions not in line with the results section of the document)

Possible conflicts of interest of the authors of the publications should also be taken into consideration.
It is recognised that, where manufacturers source clinical investigation data reported in the scientific literature, the documentation readily available to the manufacturer for inclusion in the clinical evaluation is likely to be no more than the published paper itself. In case of missing information, the rating of the methodological quality of a publication may need to be downscaled.
For additional information see Appendix A6 (Appraisal of clinical data – examples of studies that lack scientific validity for demonstration of adequate clinical performance and/or clinical safety).

9.3.2.How to determine the relevance of a data set for the clinical evaluation

When evaluating the relevance of collected data it is important to consider whether the data are intended to directly demonstrate adequate clinical performance and clinical safety of the device (often referred to as pivotal data), or whether the data serves an indirect supportive role.

1. Pivotal data
   • Pivotal data must have the data quality necessary for demonstration of adequate clinical performance and clinical safety of the device under evaluation (see Appendix A6, Appraisal of clinical data – examples of studies that lack scientific validity for demonstration of adequate clinical performance and/or clinical safety);
   • be generated either with the device under evaluation or with an equivalent device used in its intended purpose (for an equivalent device, equivalence must be demonstrated; see Appendix A1, Demonstration of equivalence).
2. Other data
   Data that are not pivotal are generally appraised and weighted for their contribution for purposes such as:

• identifying and defining the current knowledge/ state of the art in the corresponding medical field, so as to define acceptability criteria for the evaluation of the benefit/risk profile and of specific side-effects of the device under evaluation;
• identifying hazards (including hazards due to substances and technologies), individual case reports may be used for identification of new and previously unknown hazards that are associated with the device;
• justifying the validity of criteria used for the demonstration of equivalence (if equivalence is claimed);
• justifying the validity of surrogate endpoints (if surrogate endpoints are used).
• providing input for the planning of pivotal studies.

The corresponding information is, in general, summarised in a literature review section of the clinical evaluation report.

• Aspects to consider when determining relevance
  The table below shows examples of aspects that could be used for determining if and in what respect data are relevant to the clinical evaluation.

9.3.3.How to weight the contribution of each data set

Based on their scientific validity and relevance, the data should be weighted according to their relative contributions.

Due to the diversity of medical devices, there is no single, well established method for weighting clinical data:

• the evaluators should identify appropriate criteria to be applied for a specific evaluation;
• these pre-defined criteria should be followed strictly by the evaluators.

Typically, clinical data should receive the highest weighting, when generated through a well designed and monitored randomized controlled clinical investigation (also called randomised controlled trial), conducted with the device under evaluation in its intended purpose, with patients and users that are representative of the target population.

Note: It is acknowledged that randomized clinical investigations may not always be feasible and/or appropriate and the use of alternative study designs may provide relevant clinical information of adequate weighting.

When rejecting evidence, the evaluators should document the reasons (both for studies and reports that have been generated and are held by the manufacturer, and for other documents identified during Stage 1).