Experts caution on privacy and reliability of AI health tools
Health-focused AI services now let users upload medical records, but specialists warn about data security, regulatory gaps, and the potential for inaccurate advice.
A growing number of Americans are turning to artificial-intelligence platforms for health information, with a recent KFF poll showing that one-third have consulted AI about medical concerns. OpenAI recently expanded ChatGPT’s health features, allowing users to upload medical records and wearable-device data to receive personalized insights, such as diet recommendations for low-iron levels.
While many respondents expressed privacy worries, 41% of those who asked health questions said they uploaded personal data. Companies including OpenAI, Anthropic, Microsoft and Perplexity assert that conversations are encrypted, that user data is not used to train models, and that records can be deleted within 30 days. Dr. Bob Wachter, chair of UCSF’s Department of Medicine, said he trusts these promises because the firms have a strong business interest in protecting the data, but he acknowledged that the risk is not zero.
Other experts remain skeptical. Bioethicist Dr. Art Caplan warned that without comprehensive federal privacy legislation, health data could be vulnerable to breaches or secondary use, citing the 2023 23andMe breach as an example. Nephrologist Dr. Girish Nadkarni urged users to verify AI output before relying on it, and to seek independent confirmation that patient data is not used for model training.
Accuracy concerns also persist. Studies published in JAMA Oncology and Nature Medicine found that earlier AI versions gave inappropriate or down-played recommendations for cancer treatment, asthma attacks and diabetic ketoacidosis, sometimes failing to advise emergency care. OpenAI responded that newer models have improved, but researchers noted that longer interactions did not consistently enhance performance.
Industry players argue that AI can complement, not replace, clinicians. Epic’s senior vice president of R&D, Garrett Adams, described the company’s Emmie AI as an extension of the care team that translates records into plain language and escalates complex queries to human providers.
To use AI responsibly, specialists recommend limiting the duration of data sharing, framing questions without emotional language, starting new conversations for each query, and continuously asking for evidence or clarification. Ultimately, they stress that users should rely on their own judgment and seek professional medical care when urgent symptoms arise.