Clinician-curated benchmark for evaluating LLM instruction-following on real clinical tasks. Contains 983 natural-language instructions authored by clinicians, paired with 275 de-identified longitudinal EHRs from Stanford Health Care. Instructions span a wide range of clinical reasoning tasks — summarisation, differential diagnosis, care plan generation, and follow-up ordering. Designed to assess whether LLMs can faithfully execute what a clinician actually asks, not just answer medical trivia. Dataset access requires a Stanford AIMI data use agreement; evaluation code is MIT. From the Shah Lab at Stanford Medicine.
Assess — permissive licence (MIT).
Derived deterministically by the MACH scorer from the signals below — not written prose.
Dormant — last commit ~18 mo ago; no tests/CI/releases detected
★ 104 stars · 8 forks · 0 open issues
Every value records the source it was harvested from.
| Field | Value | Source |
|---|---|---|
| citations | 166 | openalex |
| description | MedAlign is a clinician-generated dataset for instruction following with electronic medical records. | github_api |
| discovered_conda | not_found | discovery |
| discovered_cran | not_found | discovery |
| discovered_doi | 10.1609/aaai.v38i20.30205,10.1609/AAAI.V38I20.30205 | discovery |
| discovered_hf | not_found | discovery |
| discovered_npm | not_found | discovery |
| discovered_pypi | not_found | discovery |
| has CI | False | github_api |
| has docs | True | github_api |
| has releases | False | github_api |
| has tests | False | github_api |
| HuggingFace downloads (monthly) | not_found | huggingface |
| judgement | Assess | mach_scorer |
| judgement reason | Assess — permissive licence (MIT). | mach_scorer |
| license | MIT | github_api |
| openalex_id | https://openalex.org/W4393157213 | openalex |
| openalex_type | conference-paper | openalex |
| openalex_venue | Proceedings of the AAAI Conference on Artificial Intelligence | openalex |
| publication_year | 2024 | openalex |
| PyPI downloads (monthly) | not_found | pypi |
| archived | False | github_api |
| repository created | 2023-08-31T17:32:02Z | github_api |
| GitHub forks | 8 | github_api |
| repo health | Dormant | mach_health |
| repo health reason | last commit ~18 mo ago; no tests/CI/releases detected | mach_health |
| homepage | ok | github_api |
| last commit | 2025-05-17T05:37:25Z | github_api |
| open issues | 0 | github_api |
| primary language | ok | github_api |
| repo_size_kb | 327 | github_api |
| GitHub stars | 104 | github_api |
| topics | benchmarking,ehr,electronic-health-records,foundation-models,healthcare,instruction-following,llms | github_api |
Recorded deviations — when a harvested value changed between runs.
| When | Field | Was | Now | Source |
|---|---|---|---|---|
| 2026-11-20T21:53:46.063208 | repo_health | — | Dormant | mach_health |
| 2026-11-20T21:53:46.063208 | repo_health_reason | — | last commit ~18 mo ago; no tests/CI/releases detected | mach_health |
| 2026-11-20T21:53:46.063151 | judgement | — | Assess | mach_scorer |
| 2026-11-20T21:53:46.063151 | judgement_reason | — | Assess — permissive licence (MIT). | mach_scorer |
| 2026-11-20T21:45:46.621795 | hf_downloads | — | — | huggingface |
| 2026-11-20T21:45:46.565285 | pypi_downloads | — | — | pypi |
| 2026-11-20T21:45:46.020263 | citations | — | 166 | openalex |
| 2026-11-20T21:45:46.020263 | openalex_id | — | https://openalex.org/W4393157213 | openalex |
| 2026-11-20T21:45:46.020263 | openalex_type | — | conference-paper | openalex |
| 2026-11-20T21:45:46.020263 | openalex_venue | — | Proceedings of the AAAI Conference on Artificial Intelligence | openalex |
| 2026-11-20T21:45:46.020263 | publication_year | — | 2024 | openalex |
| 2026-11-20T21:45:46.020216 | discovered_conda | — | — | discovery |
| 2026-11-20T21:45:46.020216 | discovered_cran | — | — | discovery |
| 2026-11-20T21:45:46.020216 | discovered_doi | — | 10.1609/aaai.v38i20.30205,10.1609/AAAI.V38I20.30205 | discovery |
| 2026-11-20T21:45:46.020216 | discovered_hf | — | — | discovery |
| 2026-11-20T21:45:46.020216 | discovered_npm | — | — | discovery |
| 2026-11-20T21:45:46.020216 | discovered_pypi | — | — | discovery |
| 2026-11-20T21:45:44.388751 | has_ci | — | False | github_api |
| 2026-11-20T21:45:44.388751 | has_docs | — | True | github_api |
| 2026-11-20T21:45:44.388751 | has_releases | — | False | github_api |
| 2026-11-20T21:45:44.388751 | has_tests | — | False | github_api |
| 2026-11-20T21:45:44.388659 | description | — | MedAlign is a clinician-generated dataset for instruction following with electronic medical records. | github_api |
| 2026-11-20T21:45:44.388659 | license | — | MIT | github_api |
| 2026-11-20T21:45:44.388659 | repo_archived | — | False | github_api |
| 2026-11-20T21:45:44.388659 | repo_created | — | 2023-08-31T17:32:02Z | github_api |
| 2026-11-20T21:45:44.388659 | repo_forks | — | 8 | github_api |
| 2026-11-20T21:45:44.388659 | repo_homepage | — | — | github_api |
| 2026-11-20T21:45:44.388659 | repo_last_commit | — | 2025-05-17T05:37:25Z | github_api |
| 2026-11-20T21:45:44.388659 | repo_open_issues | — | 0 | github_api |
| 2026-11-20T21:45:44.388659 | repo_primary_language | — | — | github_api |
| 2026-11-20T21:45:44.388659 | repo_size_kb | — | 327 | github_api |
| 2026-11-20T21:45:44.388659 | repo_stars | — | 104 | github_api |
| 2026-11-20T21:45:44.388659 | repo_topics | — | benchmarking,ehr,electronic-health-records,foundation-models,healthcare,instruction-following,llms | github_api |