In a significant experiment, a medical school in the US employed Open AI’s Chat-GPT 4 to determine its accuracy in diagnosing challenging medical cases.
Researchers at Beth Israel Deaconess Medical Center (BIDMC) in Boston discovered that Chat-GPT 4 achieved the correct diagnosis in nearly 40 per cent of cases. Moreover, the AI chatbot provided an accurate diagnosis among its list of potential diagnoses in two-thirds of the challenging cases, as per the findings published in JAMA.
Adam Rodman, co-director of the Innovations in Media and Education Delivery (iMED) Initiative at BIDMC, explained that recent advances in artificial intelligence have produced generative AI models capable of providing detailed text-based responses and excelling in standardized medical examinations. The study aimed to assess whether such a generative model could “think” like a doctor and solve complex diagnostic cases used for educational purposes, and it demonstrated impressive performance.
To evaluate the chatbot’s diagnostic abilities, Rodman and colleagues used clinicopathological case conferences (CPCs), which consist of complex patient cases with relevant clinical and laboratory data, imaging studies, and histopathological findings published for educational purposes.
In the assessment of 70 CPC cases, the AI accurately matched the final CPC diagnosis in 27 instances (39 per cent). Furthermore, in 64 per cent of the cases, the final CPC diagnosis was included in the AI’s differential, which is a list of possible conditions that could explain a patient’s symptoms, medical history, clinical findings, and lab or imaging results.
While emphasizing that chatbots cannot replace trained medical professionals, Zahir Kanjee, the first author of the study and a hospitalist at BIDMC, and assistant professor of medicine at Harvard Medical School noted that generative AI holds promise as a potential adjunct to human cognition in the diagnostic process. It has the potential to assist physicians in making sense of complex medical data and expanding or refining diagnostic thinking.
The study contributes to the growing body of evidence showcasing the potential of AI technology in healthcare. However, further research is needed to explore the optimal applications, benefits, and limitations of AI, with special attention to privacy concerns. Understanding how these new AI models might transform healthcare delivery remains an important area of investigation.
