← Back to catalog

HealthBench Adopt

Physician-curated benchmark for evaluating large language models on realistic health conversations. Contains 5,000 multi-turn conversations between a model and a user or healthcare professional, each graded by 48,562 unique rubric criteria written by 262 physicians who have practiced in 60 countries. Covers general health, emergency medicine, chronic disease, mental health, rare disease, and global health contexts. Evaluation code ships in the openai/simple-evals repository. Unlike multiple-choice benchmarks, HealthBench uses open-ended, graded rubrics to assess both accuracy and safety of model responses. Published by OpenAI in May 2025.

Judgement: Adopt

Adopt — last commit ~7 months ago; 4710 stars (strong community); 9252 monthly downloads; permissive licence (MIT).

Derived deterministically by the MACH scorer from the signals below — not written prose.

Repository

Maintained — last commit ~7 mo ago; has tests

★ 4,710 stars · 526 forks · 79 open issues

Python 60% Jupyter Notebook 40%

Signals & lineage

Every value records the source it was harvested from.

FieldValueSource
descriptionokgithub_api
discovered_condanot_founddiscovery
discovered_crannot_founddiscovery
discovered_doinot_founddiscovery
discovered_hfnot_founddiscovery
discovered_npmnot_founddiscovery
discovered_pypinot_founddiscovery
has CIFalsegithub_api
has docsTruegithub_api
has releasesFalsegithub_api
has testsTruegithub_api
HuggingFace downloads (monthly)9252huggingface
judgementAdoptmach_scorer
judgement reasonAdopt — last commit ~7 months ago; 4710 stars (strong community); 9252 monthly downloads; permissive licence (MIT).mach_scorer
licenseMITgithub_api
PyPI downloads (monthly)not_foundpypi
archivedFalsegithub_api
repository created2024-04-11T22:38:17Zgithub_api
GitHub forks526github_api
repo healthMaintainedmach_health
repo health reasonlast commit ~7 mo ago; has testsmach_health
homepageokgithub_api
languagesPython:60,Jupyter Notebook:40github_api
last commit2026-04-22T22:16:18Zgithub_api
open issues79github_api
primary languagePythongithub_api
repo_size_kb110github_api
GitHub stars4710github_api
topicsokgithub_api

Change history

Recorded deviations — when a harvested value changed between runs.

WhenFieldWasNowSource
2026-11-20T21:53:45.530556repo_health—Maintainedmach_health
2026-11-20T21:53:45.530556repo_health_reason—last commit ~7 mo ago; has testsmach_health
2026-11-20T21:53:45.530472judgement—Adoptmach_scorer
2026-11-20T21:53:45.530472judgement_reason—Adopt — last commit ~7 months ago; 4710 stars (strong community); 9252 monthly downloads; permissive licence (MIT).mach_scorer
2026-11-20T21:45:41.203851hf_downloads—9252huggingface
2026-11-20T21:45:41.147547pypi_downloads——pypi
2026-11-20T21:45:41.147496discovered_conda——discovery
2026-11-20T21:45:41.147496discovered_cran——discovery
2026-11-20T21:45:41.147496discovered_doi——discovery
2026-11-20T21:45:41.147496discovered_hf——discovery
2026-11-20T21:45:41.147496discovered_npm——discovery
2026-11-20T21:45:41.147496discovered_pypi——discovery
2026-11-20T21:45:39.608985has_ci—Falsegithub_api
2026-11-20T21:45:39.608985has_docs—Truegithub_api
2026-11-20T21:45:39.608985has_releases—Falsegithub_api
2026-11-20T21:45:39.608985has_tests—Truegithub_api
2026-11-20T21:45:39.608985repo_languages—Python:60,Jupyter Notebook:40github_api
2026-11-20T21:45:39.608915description——github_api
2026-11-20T21:45:39.608915license—MITgithub_api
2026-11-20T21:45:39.608915repo_archived—Falsegithub_api
2026-11-20T21:45:39.608915repo_created—2024-04-11T22:38:17Zgithub_api
2026-11-20T21:45:39.608915repo_forks—526github_api
2026-11-20T21:45:39.608915repo_homepage——github_api
2026-11-20T21:45:39.608915repo_last_commit—2026-04-22T22:16:18Zgithub_api
2026-11-20T21:45:39.608915repo_open_issues—79github_api
2026-11-20T21:45:39.608915repo_primary_language—Pythongithub_api
2026-11-20T21:45:39.608915repo_size_kb—110github_api
2026-11-20T21:45:39.608915repo_stars—4710github_api
2026-11-20T21:45:39.608915repo_topics——github_api