← Back to catalog

MedHELM Adopt

Comprehensive evaluation framework for benchmarking LLMs on medical tasks, built on Stanford CRFM's HELM platform. Covers 121 clinician-validated tasks across 5 domains — clinical decision support, clinical note generation, patient communication, medical research assistance, and administration/workflow — spanning 35 benchmarks (17 existing + 18 newly formulated). Uses an LLM-jury evaluation methodology with agreement rates comparable to clinician ratings. Includes a live public leaderboard at medhelm.org. Published in Nature Medicine 2025 by Stanford CRFM, Shah Lab, and Microsoft Health.

Judgement: Adopt

Adopt — committed within 3 months; 2975 stars (strong community); permissive licence (Apache-2.0).

Derived deterministically by the MACH scorer from the signals below — not written prose.

Repository

Active — last commit ~3 mo ago; has CI, releases

★ 2,975 stars · 438 forks · 128 open issues

Python 95% TypeScript 4%

Signals & lineage

Every value records the source it was harvested from.

FieldValueSource
descriptionHolistic Evaluation of Language Models (HELM) is an open source Python framework created by the Center for Research on Foundation Models (CRFM) at Stanford for holistic, reproducible and transparent evaluation of foundation models, including large language models (LLMs) and multimodal models.github_api
discovered_condanot_founddiscovery
discovered_crannot_founddiscovery
discovered_doinot_founddiscovery
discovered_hfnot_founddiscovery
discovered_npmnot_founddiscovery
discovered_pypicrfm-helmdiscovery
has CITruegithub_api
has docsTruegithub_api
has releasesTruegithub_api
has testsFalsegithub_api
HuggingFace downloads (monthly)not_foundhuggingface
judgementAdoptmach_scorer
judgement reasonAdopt — committed within 3 months; 2975 stars (strong community); permissive licence (Apache-2.0).mach_scorer
licenseApache-2.0github_api
PyPI downloads (monthly)246pypi
archivedFalsegithub_api
repository created2021-11-29T08:53:17Zgithub_api
GitHub forks438github_api
repo healthActivemach_health
repo health reasonlast commit ~3 mo ago; has CI, releasesmach_health
homepagehttps://crfm.stanford.edu/helmgithub_api
languagesPython:95,TypeScript:4github_api
last commit2026-09-01T01:33:19Zgithub_api
open issues128github_api
primary languagePythongithub_api
repo_size_kb130251github_api
GitHub stars2975github_api
topicsokgithub_api

Change history

Recorded deviations — when a harvested value changed between runs.

WhenFieldWasNowSource
2026-11-20T21:53:46.151724repo_health—Activemach_health
2026-11-20T21:53:46.151724repo_health_reason—last commit ~3 mo ago; has CI, releasesmach_health
2026-11-20T21:53:46.151647judgement—Adoptmach_scorer
2026-11-20T21:53:46.151647judgement_reason—Adopt — committed within 3 months; 2975 stars (strong community); permissive licence (Apache-2.0).mach_scorer
2026-11-20T21:45:51.119878hf_downloads——huggingface
2026-11-20T21:45:51.063552pypi_downloads—246pypi
2026-11-20T21:45:51.063496discovered_conda——discovery
2026-11-20T21:45:51.063496discovered_cran——discovery
2026-11-20T21:45:51.063496discovered_doi——discovery
2026-11-20T21:45:51.063496discovered_hf——discovery
2026-11-20T21:45:51.063496discovered_npm——discovery
2026-11-20T21:45:51.063496discovered_pypi—crfm-helmdiscovery
2026-11-20T21:45:49.412583has_ci—Truegithub_api
2026-11-20T21:45:49.412583has_docs—Truegithub_api
2026-11-20T21:45:49.412583has_releases—Truegithub_api
2026-11-20T21:45:49.412583has_tests—Falsegithub_api
2026-11-20T21:45:49.412583repo_languages—Python:95,TypeScript:4github_api
2026-11-20T21:45:49.412512description—Holistic Evaluation of Language Models (HELM) is an open source Python framework created by the Center for Research on Foundation Models (CRFM) at Stanford for holistic, reproducible and transparent evaluation of foundation models, including large language models (LLMs) and multimodal models.github_api
2026-11-20T21:45:49.412512license—Apache-2.0github_api
2026-11-20T21:45:49.412512repo_archived—Falsegithub_api
2026-11-20T21:45:49.412512repo_created—2021-11-29T08:53:17Zgithub_api
2026-11-20T21:45:49.412512repo_forks—438github_api
2026-11-20T21:45:49.412512repo_homepage—https://crfm.stanford.edu/helmgithub_api
2026-11-20T21:45:49.412512repo_last_commit—2026-09-01T01:33:19Zgithub_api
2026-11-20T21:45:49.412512repo_open_issues—128github_api
2026-11-20T21:45:49.412512repo_primary_language—Pythongithub_api
2026-11-20T21:45:49.412512repo_size_kb—130251github_api
2026-11-20T21:45:49.412512repo_stars—2975github_api
2026-11-20T21:45:49.412512repo_topics——github_api