BACKGROUND

The 21st Century Cures Act, which took effect on December 13, 2016, required FDA to issue guidance on the use of RWE in regulatory decisions. This requirement resulted in FDA’s September 30, 2021, draft guidance outlining considerations for the use of real-world data (RWD) and RWE in regulatory decisions.

Although FDA received significant feedback from industry on the draft guidance requesting more flexibility, the final version of the draft guidance largely incorporates the guidance with the following key changes:

• Clarifies that the selection of study variables for validation and the effort required for validation depend on the level of certainty required and the impact of potential misclassification on the study conclusion.
• Points out that the choice of a reference standard for validation may vary depending on the study design and question, the variable of interest and the level of certainty required.
• Recommends the use of quantitative approaches to demonstrate whether and how any misclassification might affect study results.
• Removes commonly understood, defined terms.

ANALYSIS

The guidelines address the selection of data sources, the development and validation of definitions for study design elements, and the traceability and quality of data during data collection, curation, and integration. In particular, the guidelines do not provide recommendations on study design or statistical analysis, nor do they endorse any particular type of data source or study methodology.

The guidelines use the following definitions for the most important terms:

• RWD stands for “data concerning the health status of patients or the provision of health services, routinely collected from various sources.”
• RWE stands for “clinical evidence regarding the use and potential benefit or risk of a medical device derived from the analysis of RWD”.
• Medical claims data is “information submitted to insurers for the purpose of obtaining payment for treatments and other procedures.”
• Clinical trials are understood to mean “all study designs, including in particular interventional studies in which the treatment is prescribed by a protocol and non-interventional studies in which the treatment is determined as part of routine clinical care.”

Data sources

According to the guidance, FDA recommends that protocols submitted to FDA identify all data sources proposed for the study and evaluate all of those data sources to determine their suitability for answering specific study questions. Specifically, FDA recommends that the protocol indicate the relevance and rationale for selecting applicable data sources. These rationales should take into account the likelihood that a data source contains the information sought. For example, because EHRs are collected as part of health care services and not as part of the protocol, a protocol proposing the use of EHRs should identify the type of information sought and provide relevant background information on how the EHRs collect that information.

The guidance also discusses considerations for data linkage to create broader longitudinal data sets. FDA notes that protocols should describe the accuracy and completeness of data linkages, linkage methods, and any potential issues with linkage quality through the use of probabilistic and deterministic approaches. Sponsors and stakeholders supporting such data linkage should be aware of the expectations FDA outlines in the guidance regarding linkage accuracy.

Given the proliferation of AI and its potential applications in curating unstructured data, FDA recommends that protocols proposing the use of AI or other derivation methods specify the assumptions and parameters of the algorithms, the dataset used to train and build the algorithms, and information used to monitor the algorithm and any metrics used to validate the methods. FDA clarifies that it does not endorse any type of AI technologies. It is unclear whether information about adverse events discovered during an AI scrape of the EHR would impose reporting or other obligations on the manufacturer.

Study design elements

FDA clarifies that the study should not be tailored to a specific data source, but rather that data sources should be selected to best fit the questions of interest. FDA discusses design elements and parameters such as time, study population, exposure ascertainment, and outcome ascertainment. In this discussion, FDA recommends using quantitative approaches such as quantitative bias analyses to demonstrate how important covariates might affect study results.

Data quality

The guidelines provide recommendations for reviewing the quality of EHR and health insurance data throughout the data lifecycle. FDA recommends:

• Characterization of data according to completeness, conformity and plausibility of the data values.
• Document the quality assurance and quality control (QA/QC) plan that covers transformation processes.
• Description of the procedures to ensure data integrity.

These activities, including preparation of the QA/QC plan to address data quality issues, may in practice require careful coordination between the sponsor and RWD sources, including EHR vendors and other stakeholders such as data extraction and migration providers, data tokenization providers, de-identification providers, and AI and other developers offering data curation solutions. Stakeholders should carefully review the various characteristics and issues highlighted in the guidance with regard to data quality assessment.

The guidance represents the latest step in FDA’s ongoing efforts to clarify the appropriate role of RWD and RWE in the regulatory process.

Marissa Hill Daley and Jae Hyun Lee also contributed to this article.