AI chatbots are showing promise in medicine but still fall short when it comes to diagnosing patients through conversations. While they perform well on medical exams, their accuracy drops when engaging in dynamic interactions with patients.
(https://i.imgur.com/fCCQTpw.jpeg)
These advanced AI models can ace multiple-choice tests, but they struggle with a key task for doctors: collecting information from patients and using it to make accurate diagnoses. Pranav Rajpurkar from Harvard University explains, "Large language models are impressive on tests, but they fall short in open-ended diagnostic reasoning."
Researchers tested this by simulating doctor-patient conversations. They used a method that evaluates how well clinical AI models reason and gather information. These simulated patients were based on 2,000 medical cases, mostly from U.S. medical board exams.
Shreya Johri, also at Harvard, says simulating these interactions helps test an AI's ability to take a medical history. This skill is crucial in real-life practice but can't be measured with simple case studies. The benchmark they created, called CRAFT-MD, mimics real-world scenarios where patients might not know what details are important or might only share key information when asked specific questions.
The CRAFT-MD benchmark itself uses AI. OpenAI's GPT-4 model acted as the "patient AI" during these conversations. It also graded the results by comparing the clinical AI's diagnosis to the correct answer. Human medical experts reviewed the evaluations to ensure accuracy. They also checked if the clinical AI asked the right questions and gathered all necessary information.
Even if an AI model eventually performs well on this benchmark, that doesn't mean it's better than human doctors. Rajpurkar points out that real-life medicine is much more complicated than simulations. Doctors don't just diagnose patients; they manage multiple cases, work with healthcare teams, perform physical exams, and navigate complex social and systemic challenges in healthcare systems.
"If an AI performs well on our benchmark, it could become a powerful tool to support doctors," Rajpurkar says. "But it won't replace the holistic judgment and experience of physicians."