Analysis
It is a good idea to consult a statistician already during the planning phase to enable the production of correctly analysed data and publication of the result. Early preparation is crucial for producing a good study design, the correct number of study participants and the right outcome measures to answer your research question.
Plan analysis and data management early
Analysis and data management need to be planned before a clinical study starts. The plan may be concise and included in the research plan, or take the form of a separate document, for example a statistical analysis plan (SAP) with more detailed content.
The more confirmatory the purpose of the study, the more detailed the planning should be. Exploratory studies may be described more briefly.
Analysis and study design must be aligned
In order for the study to be able to answer its research question, the following elements must be clearly aligned and readable as a coherent whole in the research plan, protocol or analysis plan:
- research question or hypothesis
- outcome measures
- study design
- analysis methods
It must be clear that what is measured can actually answer the research question.
Where relevant, the following analyses should be described in advance:
- interim analyses
- adjustments for confounders
- subgroup analyses
The level of detail should be adapted to the purpose of the study.
Studies with a confirmatory purpose
A confirmatory study is based on a hypothesis formulated before the study starts. After the study has been completed, the hypothesis should be tested using the data collected and the methods described in the research plan, protocol or analysis plan.
It is particularly important that:
- the primary outcome measure is clearly defined
- the sample size is calculated based on the primary outcome measure
- the same method used for sample size calculation is used for hypothesis testing
To minimise the risk of bias, the study should, where possible:
- include a control group (placebo or gold standard)
- use randomisation
- be double-blind
Double-blinding means that neither the study participants nor the researchers know which treatment has been administered.
Studies with an exploratory purpose
Exploratory studies aim to generate new knowledge where previous evidence is lacking. In such studies, hypothesis testing is not required.
This often means that:
- sample size does not normally need to be calculated
- a primary outcome measure does not need to be defined
- control groups, blinding and randomisation are often absent
The results are mainly reported using descriptive measures, sometimes with confidence intervals and statistical tests. The aim is to generate hypotheses for future confirmatory studies.
There are also confirmatory studies that include an exploratory component.
Data management after the study has been completed
When the practical conduct of the study has been completed and all data have been collected in a database or data file, the data should be reviewed before analysis.
Check that:
- the data are complete
- any errors or ambiguities are identified and addressed
All corrections shall be documented and retained.
Corrections and blinding
If the study is blinded, corrections must be made without unblinding the study, in order to avoid data being removed or corrected on an incorrect basis.
If the statistical analysis plan needs to be amended, this should be done before the blinding is broken. The reasons for the amendment shall be documented and clearly described when the study results are reported and published.
Amendments to the analysis plan may be considered a substantial amendment, which requires approval from the Swedish Ethical Review Authority.
If the study is not blinded, an independent party may perform corrections and any necessary data checks.
Database lock
Once data checks and corrections have been completed, the study data shall be closed or locked. This means that no further changes may be made to the raw data.
Database lock is particularly important in confirmatory studies, but may also be applied in exploratory studies.
If a database with a locking function is not available, it is important to ensure that an exact copy of the raw data file is preserved in an unchanged state.
Archiving of study data
The raw data file or database should be archived for possible future review and verification.
Analysis of study data
Before the study questions are analysed, the data may need to be processed. For example:
- variables may be constructed from collected data (for example, weight and height used to calculate BMI)
- missing values may be handled through imputation (calculations used to replace missing values in the data)
If imputation methods are used, these should be described in advance in the research plan/protocol or the statistical analysis plan.
The statistical methods used in the analysis shall be based on the research question and the study design, as described in the research plan, the protocol or the statistical analysis plan.
If the study has a primary research question, the analysis of the primary variable shall be clearly distinguished from analyses of other variables. The same applies to the presentation of the study results.
Aspects to consider in the analysis phase:
Multiplicity
If many different therapies or other interventions are to be analysed in relation to the same outcome measure, the problem with multiple comparisons should be considered, and an adjustment of the significance level may be required.
Confounders
In studies that do not include randomisation, it is important to identify and define in advance any confounders for which the main analyses should be adjusted.
A confounder is a variable that affects both the primary outcome measure and the treatment groups in the study. There are several ways to identify confounders, for example by using DAGs (Directed Acyclic Graphs).
In randomised studies, adjustments for confounders are generally not required after the fact.
Subsidiary group analyses and interactions
If an outcome measure covaries (interacts) with a variable other than the study treatment, it may be relevant to report such interactions.
The interaction variable may, for example, be:
- a variable that divides the study participants into subgroups, such as sex or age groups
- a continuous variable, such as age or weight
Whenever possible, the variables and methods planned for interaction analyses should be specified in the statistical analysis plan.
Subgroup analyses may also be exploratory in nature, for example if the analysis work reveals previously unknown interactions.
Interim analyses
Interim analyses are analyses conducted while the study is ongoing. The purpose is to provide a basis for decisions on whether the study should continue or be terminated early.
Since interim analyses affect the study sample size, it is important that they are carefully planned and described already in the research plan or the protocol.
Interim analyses are most often performed by the study’s Data Safety and Monitoring Board (DSMB), which is an independent review group. Any unblinding shall remain within the independent group. All other study personnel shall remain blinded to the results.
Unplanned interim analyses should be avoided, as they may weaken confidence in the conclusions of the study.
Points to consider
- Has the original question been answered?
- What conclusions can be drawn and what are the consequences?
- Have new questions arisen during the process?
Specific rules for medical devices
There are specific rules to attend to when it comes to clinical investigations of medical devices.
Medical devices, EU Medical Device Regulation (MDR)
Medical devices in the EU are categorized into two regulatory classes that follow different regulations. For medical devices, the EU Medical Device Regulation (MDR) applies. New regulations apply within the EU as of May 26, 2021, and additional regulations apply in Sweden from July 15, 2021.
Principles for data management and analysis
The principles of data management and analysis presented in the section above also applies for studies involving medical devices. Specific for clinical investigations of medical devices is that the standard for good clinical practice, ISO 14155:2020, shall apply. This means, among other things, that there are a number of statistical considerations to be described in the clinical investigation plan, see Annex A of the standard. For clinical investigations of medical devices, the question must also be based on the information provided by the risk management process and the clinical evaluation with regard to the clinical data that need to be produced in order to prove the safety and performance of the product. Read more about this in the step Idea. Any deviations from the investigation plan regarding the statistical aspects of the study must be evaluated with regard to whether they are to be regarded as significant changes that must be approved by the Swedish Ethical Review Authority and the Swedish Medical Products Agency before they can be implemented.
The standard for good clinical practice, ISO 14155: 2020, SIS' website External link.
Confirmatory studies of the safety and performance of medical devices are often based on the product being compared with an existing method or therapy, and in this way it can be shown that the device works equally well or better. This type of study should be randomised and blinded, to avoid bias. For studies involving medical devices, it can be difficult to design the study based on these criteria, for example because it may be impossible to develop an identical placebo or comparison device, and it may also be unsuitable to use a prospective control group for various reasons. Some of the differing ways of handling the challenges involved in medical device studies are presented below.
Medical device with a therapeutic purpose
Control group: If it is possible to use a prospective control group that is treated with a standard therapy, then this is to be preferred. Otherwise, it may be possible to use a historical control, where you know that a certain therapy or lack of therapy has a certain outcome, which can then be compared with the outcome of the study.
Randomising and blinding: If you use a historical control, then you do of course lose the opportunity of randomising and blinding the therapy. But if you use a prospective control group, it can still be difficult to keep patients and the physicians providing treatment blinded, if it has not been possible to develop a placebo therapy. In this case, an independent evaluating physician may conduct a follow-up evaluation of the therapy result in a blinded way (that is to say, without knowing what therapy the patient has been randomised to).
Medical device with a diagnostic purpose
Control group: Often there is a standard method for making a diagnosis in the area. This method has a known ability to make a specific diagnosis (known as sensitivity and specificity, which can be used for comparison with the results of the trial device.
Randomising and blinding: In some cases, patients in diagnostic studies can be their own control, that is to say that they are subjected to two types of investigation, where one is an investigation using the standard method, and one is an investigation using the trial device. To increase safety for the study participant, the patient is diagnosed using the standard method. An independent physician can, however, make a diagnosis using the standard method and the trial device in a blinded and randomised way, to see whether the diagnoses generated by each method correspond.
As for all clinical studies, it is recomended that the study design, data management and analysis are discussed with a statistician as necessary. ICH E9 Statistical Principles for Clinical Trials is a document primarily produced for clinical medicine studies, but the statistical principles can be applied for medical device studies as well.
ICH E9 Statistical Principles for Clinical Trials, ICH website External link.
ECRIN has produced a database of outcome measurements that have been used in studies of different types of medical devices. Here you can get tips of how others have chosen to measure the effect and performance of medical devices. Harmonisation of outcome measurements can also facilitate future meta-analyses and summaries of the evidence situation for devices that have been evaluated in the same way.
Database of outcome measurements, ECRIN website External link.
Medical devices for in vitro diagnostics, EU regulation IVDR
Medical devices in the EU are regulated and divided into two categories that follow different frameworks. For in vitro diagnostic medical devices, the EU Regulation 2017/746 on in vitro diagnostic medical devices (IVDR) applies. New regulations came into effect within the EU from May 26, 2022.
Principles for data management and analysis
The principles for data management and analysis presented in the general section above also apply to medical technology studies. Specific to in vitro diagnostic products is that the standard of good clinical practice ISO 20916: 2019 should be applied for clinical performance studies. This means, among other things, that there are a number of statistical considerations that must be described in the clinical trial plan, according to the standard. The purpose of these studies is to evaluate analytical or clinical performance.
Analytical performance: the ability of a product to accurately detect or measure an analyte.
Clinical performance: the ability of a product to provide accurate medical information such as diagnosis.
The standard for good clinical practice, ISO 20916:2019, SIS' website External link.
To evaluate the clinical performance, the medical information collected with the evaluated IVD product can be compared with medical information obtained through the use of other diagnostic techniques.
Also read under the heading Medical device with diagnostic purpose above.
As with all clinical studies, it is recommended that study arrangements, data management and analysis be discussed with a statistician if necessary.
Related information about ATMP
ATMP (Advanced therapy medicinal products)
Advanced therapy medicinal products (ATMP) are human medicines, based on genes, tissues or human cells. They offer ground-breaking new opportunities for the treatment of diseases and injuries. More information about advanced therapy:
For guide/templates and regulatory guide, ATMP project website External link.
Classifications, regulatory information, European Medicines Agency website External link.
Published: