DeeperML - #patients

OpenAI has introduced HealthBench, a new open-source benchmark designed to evaluate AI performance in realistic healthcare scenarios. Developed in collaboration with over 262 physicians, HealthBench uses 5,000 multi-turn conversations and over 48,000 rubric criteria to grade AI models across seven medical domains and 49 languages. The benchmark assesses AI responses based on communication quality, instruction following, accuracy, contextual understanding, and completeness, providing a comprehensive evaluation of AI capabilities in healthcare. OpenAI’s latest models, including o3 and GPT-4.1, have shown impressive results on this benchmark.

The most provocative finding from the HealthBench evaluation is that the newest AI models are performing at or beyond the level of human experts in crafting responses to medical queries. Earlier tests from September 2024 showed that doctors could improve AI outputs by editing them, scoring higher than doctors working without AI. However, with the latest April 2025 models, like o3 and GPT-4.1, physicians using these AI responses as a base, on average, did not further improve them. This suggests that for the specific task of generating HealthBench responses, the newest AI matches or exceeds the capabilities of human experts, even with a strong AI starting point.

In related news, FaceAge, a face-reading AI tool developed by researchers at Mass General Brigham, demonstrates promising abilities in predicting cancer outcomes. By analyzing facial photographs, FaceAge estimates a person's biological age and can predict cancer survival with an impressive 81% accuracy rate. This outperforms clinicians in predicting short-term life expectancy, especially for patients receiving palliative radiotherapy. FaceAge identifies subtle facial features associated with aging and provides a quantifiable measure of biological aging that correlates with survival outcomes and health risks, offering doctors more objective and precise survival estimates.


References :

• pub.towardsai.net: This week, OpenAI unveiled HealthBench, a significant new open-source benchmark evaluating AI in realistic healthcare scenarios.
• www.marktechpost.com: This news piece mentions the HealthBench benchmark for evaluating AI models in healthcare.
• the-decoder.com: The article refers to the HealthBench benchmark developed by OpenAI to assess AI's capabilities in handling healthcare scenarios.
• www.analyticsvidhya.com: This blog post reports on the release of OpenAI’s HealthBench, an open-source benchmark for evaluating AI models in healthcare.
• THE DECODER: OpenAI says its latest models outperform doctors in medical benchmark
• www.zdnet.com: OpenAI's HealthBench shows AI's medical advice is improving - but who will listen?
• MarkTechPost: OpenAI Releases HealthBench: An Open-Source Benchmark for Measuring the Performance and Safety of Large Language Models in Healthcare
• eWEEK: FaceAge, a face-reading AI tool that estimates biological age from facial photographs, predicts cancer outcomes with an impressive 81% accuracy rate.
• The Rundown AI: PLUS: OpenAI launches HealthBench to evaluate AI in healthcare
• the-decoder.com: The article discusses OpenAI's HealthBench benchmark for evaluating large language models in realistic healthcare settings.
• www.eweek.com: FaceAge AI Tool Surpasses Doctors with 81% Accuracy in Cancer Survival Prediction
• Fello AI: Forget everything you thought you knew about medicine! Artificial Intelligence is crashing into healthcare with the force of a meteor, and the breakthroughs are coming so fast it’s hard to keep up.
• Microsoft Research: Peter Lee and his coauthors, Carey Goldberg and Dr. Zak Kohane, reflect on how generative AI is unfolding in real-world healthcare, drawing on earlier guest conversations to examine what’s working, what’s not, and what questions still remain. The post appeared first on .

Classification:

• HashTags: #AIinHealthcare #MedicalAI #AIvsDoctors
• Target: Patients
• Product: AMIE, FaceAge, o3
• Feature: AI in Healthcare
• Type: AI
• Severity: Medium