Comprehensive evaluation framework for benchmarking LLMs on medical tasks, built on Stanford CRFM's HELM platform. Covers 121 clinician-validated tasks across 5 domains — clinical decision support, clinical note generation, patient communication, medical research assistance, and administration/workflow — spanning 35 benchmarks (17 existing + 18 newly formulated). Uses an LLM-jury evaluation methodology with agreement rates comparable to clinician ratings. Includes a live public leaderboard at medhelm.org. Published in Nature Medicine 2025 by Stanford CRFM, Shah Lab, and Microsoft Health.
Adopt — committed within 3 months; 2975 stars (strong community); permissive licence (Apache-2.0).
Derived deterministically by the MACH scorer from the signals below — not written prose.
Active — last commit ~3 mo ago; has CI, releases
★ 2,975 stars · 438 forks · 128 open issues
Python 95% TypeScript 4%
Every value records the source it was harvested from.
| Field | Value | Source |
|---|---|---|
| description | Holistic Evaluation of Language Models (HELM) is an open source Python framework created by the Center for Research on Foundation Models (CRFM) at Stanford for holistic, reproducible and transparent evaluation of foundation models, including large language models (LLMs) and multimodal models. | github_api |
| discovered_conda | not_found | discovery |
| discovered_cran | not_found | discovery |
| discovered_doi | not_found | discovery |
| discovered_hf | not_found | discovery |
| discovered_npm | not_found | discovery |
| discovered_pypi | crfm-helm | discovery |
| has CI | True | github_api |
| has docs | True | github_api |
| has releases | True | github_api |
| has tests | False | github_api |
| HuggingFace downloads (monthly) | not_found | huggingface |
| judgement | Adopt | mach_scorer |
| judgement reason | Adopt — committed within 3 months; 2975 stars (strong community); permissive licence (Apache-2.0). | mach_scorer |
| license | Apache-2.0 | github_api |
| PyPI downloads (monthly) | 246 | pypi |
| archived | False | github_api |
| repository created | 2021-11-29T08:53:17Z | github_api |
| GitHub forks | 438 | github_api |
| repo health | Active | mach_health |
| repo health reason | last commit ~3 mo ago; has CI, releases | mach_health |
| homepage | https://crfm.stanford.edu/helm | github_api |
| languages | Python:95,TypeScript:4 | github_api |
| last commit | 2026-09-01T01:33:19Z | github_api |
| open issues | 128 | github_api |
| primary language | Python | github_api |
| repo_size_kb | 130251 | github_api |
| GitHub stars | 2975 | github_api |
| topics | ok | github_api |
Recorded deviations — when a harvested value changed between runs.
| When | Field | Was | Now | Source |
|---|---|---|---|---|
| 2026-11-20T21:53:46.151724 | repo_health | — | Active | mach_health |
| 2026-11-20T21:53:46.151724 | repo_health_reason | — | last commit ~3 mo ago; has CI, releases | mach_health |
| 2026-11-20T21:53:46.151647 | judgement | — | Adopt | mach_scorer |
| 2026-11-20T21:53:46.151647 | judgement_reason | — | Adopt — committed within 3 months; 2975 stars (strong community); permissive licence (Apache-2.0). | mach_scorer |
| 2026-11-20T21:45:51.119878 | hf_downloads | — | — | huggingface |
| 2026-11-20T21:45:51.063552 | pypi_downloads | — | 246 | pypi |
| 2026-11-20T21:45:51.063496 | discovered_conda | — | — | discovery |
| 2026-11-20T21:45:51.063496 | discovered_cran | — | — | discovery |
| 2026-11-20T21:45:51.063496 | discovered_doi | — | — | discovery |
| 2026-11-20T21:45:51.063496 | discovered_hf | — | — | discovery |
| 2026-11-20T21:45:51.063496 | discovered_npm | — | — | discovery |
| 2026-11-20T21:45:51.063496 | discovered_pypi | — | crfm-helm | discovery |
| 2026-11-20T21:45:49.412583 | has_ci | — | True | github_api |
| 2026-11-20T21:45:49.412583 | has_docs | — | True | github_api |
| 2026-11-20T21:45:49.412583 | has_releases | — | True | github_api |
| 2026-11-20T21:45:49.412583 | has_tests | — | False | github_api |
| 2026-11-20T21:45:49.412583 | repo_languages | — | Python:95,TypeScript:4 | github_api |
| 2026-11-20T21:45:49.412512 | description | — | Holistic Evaluation of Language Models (HELM) is an open source Python framework created by the Center for Research on Foundation Models (CRFM) at Stanford for holistic, reproducible and transparent evaluation of foundation models, including large language models (LLMs) and multimodal models. | github_api |
| 2026-11-20T21:45:49.412512 | license | — | Apache-2.0 | github_api |
| 2026-11-20T21:45:49.412512 | repo_archived | — | False | github_api |
| 2026-11-20T21:45:49.412512 | repo_created | — | 2021-11-29T08:53:17Z | github_api |
| 2026-11-20T21:45:49.412512 | repo_forks | — | 438 | github_api |
| 2026-11-20T21:45:49.412512 | repo_homepage | — | https://crfm.stanford.edu/helm | github_api |
| 2026-11-20T21:45:49.412512 | repo_last_commit | — | 2026-09-01T01:33:19Z | github_api |
| 2026-11-20T21:45:49.412512 | repo_open_issues | — | 128 | github_api |
| 2026-11-20T21:45:49.412512 | repo_primary_language | — | Python | github_api |
| 2026-11-20T21:45:49.412512 | repo_size_kb | — | 130251 | github_api |
| 2026-11-20T21:45:49.412512 | repo_stars | — | 2975 | github_api |
| 2026-11-20T21:45:49.412512 | repo_topics | — | — | github_api |