Study Finds ChatGPT Health Underestimates Severity in Over Half of Medical Emergencies

A recent study published in Nature Medicine indicates that ChatGPT Health, OpenAI's health-focused chatbot, underestimated the severity of medical emergencies in 51.6% of cases. Researchers presented 60 real-life medical scenarios to the chatbot, comparing its triage recommendations to those of three physicians. The chatbot frequently suggested patients seek medical attention within 24 to 48 hours instead of recommending immediate emergency care.

Dr. Ashwin Ramaswamy, the study's lead author and an instructor of urology at The Mount Sinai Hospital in New York City, noted that in cases like impending respiratory failure, the bot seemed to be "waiting for the emergency to become undeniable" before recommending the ER. He described the bot's behavior as "paradoxical" and "inverted to clinical risk."

OpenAI responded, stating that the study did not reflect typical use of ChatGPT Health, emphasizing that the chatbot is designed for users to provide more context through follow-up questions. The company is working to improve the model's safety and reliability before wider release.

The study also found that ChatGPT Health over-triaged 64.8% of nonurgent cases, recommending doctor visits when at-home care was sufficient. In scenarios involving suicidal ideation, the chatbot's responses were inconsistent, sometimes failing to refer users to appropriate crisis resources.

Experts caution against relying solely on AI for medical advice. Dr. John Mafi, an associate professor of medicine at UCLA Health, emphasized the need for rigorous testing before deploying such tools for life-affecting decisions. Dr. Ethan Goh, executive director of ARISE, an AI research network, stated that while AI can be helpful, it is not a substitute for a physician's advice.