← Back to catalog

MedAlign Assess

Clinician-curated benchmark for evaluating LLM instruction-following on real clinical tasks. Contains 983 natural-language instructions authored by clinicians, paired with 275 de-identified longitudinal EHRs from Stanford Health Care. Instructions span a wide range of clinical reasoning tasks — summarisation, differential diagnosis, care plan generation, and follow-up ordering. Designed to assess whether LLMs can faithfully execute what a clinician actually asks, not just answer medical trivia. Dataset access requires a Stanford AIMI data use agreement; evaluation code is MIT. From the Shah Lab at Stanford Medicine.

Judgement: Assess

Assess — permissive licence (MIT).

Derived deterministically by the MACH scorer from the signals below — not written prose.

Repository

Dormant — last commit ~18 mo ago; no tests/CI/releases detected

★ 104 stars · 8 forks · 0 open issues

Signals & lineage

Every value records the source it was harvested from.

FieldValueSource
citations166openalex
descriptionMedAlign is a clinician-generated dataset for instruction following with electronic medical records.github_api
discovered_condanot_founddiscovery
discovered_crannot_founddiscovery
discovered_doi10.1609/aaai.v38i20.30205,10.1609/AAAI.V38I20.30205discovery
discovered_hfnot_founddiscovery
discovered_npmnot_founddiscovery
discovered_pypinot_founddiscovery
has CIFalsegithub_api
has docsTruegithub_api
has releasesFalsegithub_api
has testsFalsegithub_api
HuggingFace downloads (monthly)not_foundhuggingface
judgementAssessmach_scorer
judgement reasonAssess — permissive licence (MIT).mach_scorer
licenseMITgithub_api
openalex_idhttps://openalex.org/W4393157213openalex
openalex_typeconference-paperopenalex
openalex_venueProceedings of the AAAI Conference on Artificial Intelligenceopenalex
publication_year2024openalex
PyPI downloads (monthly)not_foundpypi
archivedFalsegithub_api
repository created2023-08-31T17:32:02Zgithub_api
GitHub forks8github_api
repo healthDormantmach_health
repo health reasonlast commit ~18 mo ago; no tests/CI/releases detectedmach_health
homepageokgithub_api
last commit2025-05-17T05:37:25Zgithub_api
open issues0github_api
primary languageokgithub_api
repo_size_kb327github_api
GitHub stars104github_api
topicsbenchmarking,ehr,electronic-health-records,foundation-models,healthcare,instruction-following,llmsgithub_api

Change history

Recorded deviations — when a harvested value changed between runs.

WhenFieldWasNowSource
2026-11-20T21:53:46.063208repo_health—Dormantmach_health
2026-11-20T21:53:46.063208repo_health_reason—last commit ~18 mo ago; no tests/CI/releases detectedmach_health
2026-11-20T21:53:46.063151judgement—Assessmach_scorer
2026-11-20T21:53:46.063151judgement_reason—Assess — permissive licence (MIT).mach_scorer
2026-11-20T21:45:46.621795hf_downloads——huggingface
2026-11-20T21:45:46.565285pypi_downloads——pypi
2026-11-20T21:45:46.020263citations—166openalex
2026-11-20T21:45:46.020263openalex_id—https://openalex.org/W4393157213openalex
2026-11-20T21:45:46.020263openalex_type—conference-paperopenalex
2026-11-20T21:45:46.020263openalex_venue—Proceedings of the AAAI Conference on Artificial Intelligenceopenalex
2026-11-20T21:45:46.020263publication_year—2024openalex
2026-11-20T21:45:46.020216discovered_conda——discovery
2026-11-20T21:45:46.020216discovered_cran——discovery
2026-11-20T21:45:46.020216discovered_doi—10.1609/aaai.v38i20.30205,10.1609/AAAI.V38I20.30205discovery
2026-11-20T21:45:46.020216discovered_hf——discovery
2026-11-20T21:45:46.020216discovered_npm——discovery
2026-11-20T21:45:46.020216discovered_pypi——discovery
2026-11-20T21:45:44.388751has_ci—Falsegithub_api
2026-11-20T21:45:44.388751has_docs—Truegithub_api
2026-11-20T21:45:44.388751has_releases—Falsegithub_api
2026-11-20T21:45:44.388751has_tests—Falsegithub_api
2026-11-20T21:45:44.388659description—MedAlign is a clinician-generated dataset for instruction following with electronic medical records.github_api
2026-11-20T21:45:44.388659license—MITgithub_api
2026-11-20T21:45:44.388659repo_archived—Falsegithub_api
2026-11-20T21:45:44.388659repo_created—2023-08-31T17:32:02Zgithub_api
2026-11-20T21:45:44.388659repo_forks—8github_api
2026-11-20T21:45:44.388659repo_homepage——github_api
2026-11-20T21:45:44.388659repo_last_commit—2025-05-17T05:37:25Zgithub_api
2026-11-20T21:45:44.388659repo_open_issues—0github_api
2026-11-20T21:45:44.388659repo_primary_language——github_api
2026-11-20T21:45:44.388659repo_size_kb—327github_api
2026-11-20T21:45:44.388659repo_stars—104github_api
2026-11-20T21:45:44.388659repo_topics—benchmarking,ehr,electronic-health-records,foundation-models,healthcare,instruction-following,llmsgithub_api