Physician-curated benchmark for evaluating large language models on realistic health conversations. Contains 5,000 multi-turn conversations between a model and a user or healthcare professional, each graded by 48,562 unique rubric criteria written by 262 physicians who have practiced in 60 countries. Covers general health, emergency medicine, chronic disease, mental health, rare disease, and global health contexts. Evaluation code ships in the openai/simple-evals repository. Unlike multiple-choice benchmarks, HealthBench uses open-ended, graded rubrics to assess both accuracy and safety of model responses. Published by OpenAI in May 2025.
Adopt — last commit ~7 months ago; 4710 stars (strong community); 9252 monthly downloads; permissive licence (MIT).
Derived deterministically by the MACH scorer from the signals below — not written prose.
Maintained — last commit ~7 mo ago; has tests
★ 4,710 stars · 526 forks · 79 open issues
Python 60% Jupyter Notebook 40%
Every value records the source it was harvested from.
| Field | Value | Source |
|---|---|---|
| description | ok | github_api |
| discovered_conda | not_found | discovery |
| discovered_cran | not_found | discovery |
| discovered_doi | not_found | discovery |
| discovered_hf | not_found | discovery |
| discovered_npm | not_found | discovery |
| discovered_pypi | not_found | discovery |
| has CI | False | github_api |
| has docs | True | github_api |
| has releases | False | github_api |
| has tests | True | github_api |
| HuggingFace downloads (monthly) | 9252 | huggingface |
| judgement | Adopt | mach_scorer |
| judgement reason | Adopt — last commit ~7 months ago; 4710 stars (strong community); 9252 monthly downloads; permissive licence (MIT). | mach_scorer |
| license | MIT | github_api |
| PyPI downloads (monthly) | not_found | pypi |
| archived | False | github_api |
| repository created | 2024-04-11T22:38:17Z | github_api |
| GitHub forks | 526 | github_api |
| repo health | Maintained | mach_health |
| repo health reason | last commit ~7 mo ago; has tests | mach_health |
| homepage | ok | github_api |
| languages | Python:60,Jupyter Notebook:40 | github_api |
| last commit | 2026-04-22T22:16:18Z | github_api |
| open issues | 79 | github_api |
| primary language | Python | github_api |
| repo_size_kb | 110 | github_api |
| GitHub stars | 4710 | github_api |
| topics | ok | github_api |
Recorded deviations — when a harvested value changed between runs.
| When | Field | Was | Now | Source |
|---|---|---|---|---|
| 2026-11-20T21:53:45.530556 | repo_health | — | Maintained | mach_health |
| 2026-11-20T21:53:45.530556 | repo_health_reason | — | last commit ~7 mo ago; has tests | mach_health |
| 2026-11-20T21:53:45.530472 | judgement | — | Adopt | mach_scorer |
| 2026-11-20T21:53:45.530472 | judgement_reason | — | Adopt — last commit ~7 months ago; 4710 stars (strong community); 9252 monthly downloads; permissive licence (MIT). | mach_scorer |
| 2026-11-20T21:45:41.203851 | hf_downloads | — | 9252 | huggingface |
| 2026-11-20T21:45:41.147547 | pypi_downloads | — | — | pypi |
| 2026-11-20T21:45:41.147496 | discovered_conda | — | — | discovery |
| 2026-11-20T21:45:41.147496 | discovered_cran | — | — | discovery |
| 2026-11-20T21:45:41.147496 | discovered_doi | — | — | discovery |
| 2026-11-20T21:45:41.147496 | discovered_hf | — | — | discovery |
| 2026-11-20T21:45:41.147496 | discovered_npm | — | — | discovery |
| 2026-11-20T21:45:41.147496 | discovered_pypi | — | — | discovery |
| 2026-11-20T21:45:39.608985 | has_ci | — | False | github_api |
| 2026-11-20T21:45:39.608985 | has_docs | — | True | github_api |
| 2026-11-20T21:45:39.608985 | has_releases | — | False | github_api |
| 2026-11-20T21:45:39.608985 | has_tests | — | True | github_api |
| 2026-11-20T21:45:39.608985 | repo_languages | — | Python:60,Jupyter Notebook:40 | github_api |
| 2026-11-20T21:45:39.608915 | description | — | — | github_api |
| 2026-11-20T21:45:39.608915 | license | — | MIT | github_api |
| 2026-11-20T21:45:39.608915 | repo_archived | — | False | github_api |
| 2026-11-20T21:45:39.608915 | repo_created | — | 2024-04-11T22:38:17Z | github_api |
| 2026-11-20T21:45:39.608915 | repo_forks | — | 526 | github_api |
| 2026-11-20T21:45:39.608915 | repo_homepage | — | — | github_api |
| 2026-11-20T21:45:39.608915 | repo_last_commit | — | 2026-04-22T22:16:18Z | github_api |
| 2026-11-20T21:45:39.608915 | repo_open_issues | — | 79 | github_api |
| 2026-11-20T21:45:39.608915 | repo_primary_language | — | Python | github_api |
| 2026-11-20T21:45:39.608915 | repo_size_kb | — | 110 | github_api |
| 2026-11-20T21:45:39.608915 | repo_stars | — | 4710 | github_api |
| 2026-11-20T21:45:39.608915 | repo_topics | — | — | github_api |