Artificial intelligence (AI) has virtually limitless applications in healthcare, from automatically generating patient messages in MyChart to optimizing organ transplants and more precise tumor removal. Despite their potential benefits for doctors and patients, these tools face skepticism due to concerns about patient privacy, potential bias, and device accuracy.

In response to the rapidly evolving use and approval of AI-based medical devices in healthcare, a multi-institutional team of researchers from the UNC School of Medicine, Duke University, Ally Bank, Oxford University, Colombia University and the University of Miami set out to build public trust and evaluate how exactly AI and algorithmic technologies will be approved for use in patient care.

Together, Sammy Chouffani El Fassi, MD candidate at the UNC School of Medicine and research fellow at the Duke Heart Center, and Gail E. Henderson, Ph.D., professor in the UNC Department of Social Medicine, conducted a thorough analysis of clinical validation data for more than 500 medical AI devices. They found that about half of the tools approved by the U.S. Food and Drug Administration (FDA) lacked clinical validation data.

Their results were published in Natural medicine.

“Although AI device manufacturers boast about the credibility of their technology and FDA approval, approval does not mean that the devices have been properly tested for clinical effectiveness using real patient data,” said Chouffani El Fassi, the paper’s lead author.

“With these findings, we hope to encourage the FDA and industry to strengthen the credibility of device approval by conducting clinical validation studies of these technologies and making the results of such studies publicly available.”

Since 2016, the average number of FDA approvals of AI medical devices per year has increased from two to 69, indicating tremendous growth in the commercialization of AI medical technologies. The majority of approved AI medical technologies are used to assist physicians in diagnosing abnormalities in radiological imaging, pathological slide analysis, drug dosing, and predicting disease progression.

Artificial intelligence is able to learn and perform such human-like functions by using combinations of algorithms. The technology is then given a wealth of data and sets of rules to follow, allowing it to “learn” to recognize patterns and relationships effortlessly.

From there, device manufacturers must ensure that the technology not only stores the data the AI ​​was previously trained with, but that it can deliver accurate results using completely new solutions.

Regulation during the rapid proliferation of AI medical devices

Following the rapid proliferation of these devices and applications at the FDA, Chouffani El Fassi and Henderson et al. were curious to see how clinically effective and safe the approved devices were. Their team analyzed all submissions available in the FDA’s official database entitled “Medical Devices Using Artificial Intelligence and Machine Learning (AI/ML).”

“Many of the devices that came to market after 2016 were newly developed or perhaps similar to a product that was already on the market,” Henderson said. “With these hundreds of devices in this database, we wanted to find out what it really means when an AI medical device is approved by the FDA.”

Of the 521 device approvals, 144 were “retrospectively validated,” 148 were “prospectively validated,” and 22 were validated by randomized controlled trials. Of particular note, 226 of 521 FDA-approved medical devices (approximately 43%) lacked published clinical validation data.

Some of the devices used “phantom images” or computer-generated images that did not come from a real patient and therefore did not technically meet the requirements for clinical validation.

In addition, the researchers noted that the FDA’s most recent draft guidance, published in September 2023, does not clearly distinguish between different types of clinical validation studies in its recommendations to manufacturers.

Types of clinical validation and a new standard

In the field of clinical validation, there are three different methods that researchers and device manufacturers use to validate the accuracy of their technologies: retrospective validation, prospective validation, and a subset of prospective validation called randomized controlled trials.

Retrospective validation involves feeding the AI ​​model with image data from the past, such as chest X-rays of patients before the COVID-19 pandemic.

However, prospective validation typically provides stronger scientific evidence because the AI ​​device is validated using real-time data from patients. This is more realistic, according to the researchers, as it allows the AI ​​to take into account data variables that were not present at the time of training, such as chest X-rays of patients affected by viruses during the COVID pandemic.

Randomized controlled trials are considered the gold standard for clinical validation. This type of prospective study uses random assignment controls for confounding variables that distinguish the experimental and control groups, thus isolating the therapeutic effect of the device.

For example, researchers could evaluate the performance of devices by randomly having patients’ CT scans evaluated by a radiologist (control group) or an AI (experimental group).

Because retrospective studies, prospective studies, and randomized controlled trials provide different levels of scientific evidence, researchers involved in the study recommend that the FDA and device manufacturers clearly distinguish between the different types of clinical validation studies in their recommendations to manufacturers.

In their Natural medicine In their publication, Chouffani El Fassi, Henderson and others provide definitions for clinical validation methods that can be used as standards in the field of medical AI.

“We have shared our findings with FDA directors responsible for regulating medical devices, and we expect our work will help them in their regulatory decisions,” said Chouffani El Fassi.

“We also hope that our publication will inspire researchers and universities worldwide to conduct clinical validation studies on medical AI to improve the safety and effectiveness of these technologies. We look forward to the positive impact this project will have on patient care at scale.”

Algorithms can save lives

Chouffani El Fassi is currently working with UNC cardiothoracic surgeons Aurelie Merlo and Benjamin Haithcock and the UNC Health leadership team to implement an algorithm into their electronic health record system that will automate the organ donor evaluation and referral process.

In contrast to the rapid development of AI devices in this field, medicine lacks basic algorithms, such as computer software that diagnoses patients based on simple laboratory values ​​from electronic health records. According to Chouffani El Fassi, this is because implementation is often expensive and requires interdisciplinary teams with expertise in both medicine and computer science.

Despite these challenges, UNC Health’s goal is to improve the field of organ transplantation.

“Finding a potential organ donor, examining their organs and then arranging an organ transplant with the organ procurement organization is a lengthy and complicated process,” says Chouffani El Fassi.

“If this very simple computer algorithm works, we could streamline the process of organ donation. A single additional donor means multiple lives saved. With such a low threshold for success, we look forward to giving more people a second chance at life.”