THURSDAY, Aug. 8, 2024 (HealthDay News) — ChatGPT is not accurate as a diagnostic tool, but it offers some medical-educational benefits, according to a study published online July 31 in Plus One.

Ali Hadi, from the Schulich School of Medicine and Dentistry at Western University in London, Ontario, Canada, and colleagues investigated the diagnostic accuracy and utility of ChatGPT in medical education. The analysis included 150 Medscape case challenges (September 2021 to January 2023) entered into ChatGPT.

The researchers found that ChatGPT answered 49 percent of cases correctly, with an overall accuracy of 74 percent, a precision of 48.67 percent, a sensitivity of 48.67 percent, a specificity of 82.89 percent, and an area under the curve of 0.66. ChatGPT struggled with the interpretation of laboratory values ​​and imaging results, but generally correctly ruled out a specific differential diagnosis and provided reasonable next diagnostic steps. Just over half of the responses were complete and relevant (52 percent), and a similar percentage of responses were characterized as low cognitive load (51 percent).

“Further research should focus on improving the accuracy and reliability of ChatGPT as a diagnostic tool,” the authors write. “Integrating ChatGPT into medical education and clinical practice requires a thorough investigation of its educational and clinical limitations. Transparent guidelines for the clinical use of ChatGPT should be established and medical students and clinicians should be trained in the effective and responsible use of the tool.”

Summary/Full text

Copyright © 2024 HealthDay. All rights reserved.