FHIR-native benchmark for evaluating AI agents on real-world clinical tasks authored by practising physicians. Contains 300 tasks across 10 clinical specialties, 100 synthetic patient profiles, and over 700,000 FHIR data points. Tasks mirror genuine EHR workflows: ordering labs, interpreting results, writing referrals, and medication management. Published in NEJM AI 2025.
Assess — last commit ~12 months ago; 352 stars; permissive licence (MIT).
Derived deterministically by the MACH scorer from the signals below — not written prose.
Maintained — last commit ~12 mo ago; no tests/CI/releases detected
★ 352 stars · 72 forks · 7 open issues
Python 77% HTML 23%
Every value records the source it was harvested from.
| Field | Value | Source |
|---|---|---|
| citations | 314 | openalex |
| description | MedAgentBench: A Realistic Virtual EHR Environment to Benchmark Medical LLM Agents | github_api |
| discovered_conda | not_found | discovery |
| discovered_cran | not_found | discovery |
| discovered_doi | 10.1056/AIdbp2500144 | discovery |
| discovered_hf | not_found | discovery |
| discovered_npm | not_found | discovery |
| discovered_pypi | not_found | discovery |
| has CI | False | github_api |
| has docs | True | github_api |
| has releases | False | github_api |
| has tests | False | github_api |
| HuggingFace downloads (monthly) | not_found | huggingface |
| judgement | Assess | mach_scorer |
| judgement reason | Assess — last commit ~12 months ago; 352 stars; permissive licence (MIT). | mach_scorer |
| license | MIT | github_api |
| openalex_id | https://openalex.org/W4413801638 | openalex |
| openalex_type | article | openalex |
| openalex_venue | NEJM AI | openalex |
| publication_year | 2025 | openalex |
| PyPI downloads (monthly) | not_found | pypi |
| archived | False | github_api |
| repository created | 2025-01-22T17:14:58Z | github_api |
| GitHub forks | 72 | github_api |
| repo health | Maintained | mach_health |
| repo health reason | last commit ~12 mo ago; no tests/CI/releases detected | mach_health |
| homepage | https://ai.nejm.org/doi/full/10.1056/AIdbp2500144 | github_api |
| languages | Python:77,HTML:23 | github_api |
| last commit | 2025-11-21T18:39:26Z | github_api |
| open issues | 7 | github_api |
| primary language | Python | github_api |
| repo_size_kb | 1055 | github_api |
| GitHub stars | 352 | github_api |
| topics | agent,benchmark,medical | github_api |
Recorded deviations — when a harvested value changed between runs.
| When | Field | Was | Now | Source |
|---|---|---|---|---|
| 2026-11-20T21:53:46.041009 | repo_health | — | Maintained | mach_health |
| 2026-11-20T21:53:46.041009 | repo_health_reason | — | last commit ~12 mo ago; no tests/CI/releases detected | mach_health |
| 2026-11-20T21:53:46.040948 | judgement | — | Assess | mach_scorer |
| 2026-11-20T21:53:46.040948 | judgement_reason | — | Assess — last commit ~12 months ago; 352 stars; permissive licence (MIT). | mach_scorer |
| 2026-11-20T21:45:43.988506 | hf_downloads | — | — | huggingface |
| 2026-11-20T21:45:43.931665 | pypi_downloads | — | — | pypi |
| 2026-11-20T21:45:43.409435 | citations | — | 314 | openalex |
| 2026-11-20T21:45:43.409435 | openalex_id | — | https://openalex.org/W4413801638 | openalex |
| 2026-11-20T21:45:43.409435 | openalex_type | — | article | openalex |
| 2026-11-20T21:45:43.409435 | openalex_venue | — | NEJM AI | openalex |
| 2026-11-20T21:45:43.409435 | publication_year | — | 2025 | openalex |
| 2026-11-20T21:45:43.409388 | discovered_conda | — | — | discovery |
| 2026-11-20T21:45:43.409388 | discovered_cran | — | — | discovery |
| 2026-11-20T21:45:43.409388 | discovered_doi | — | 10.1056/AIdbp2500144 | discovery |
| 2026-11-20T21:45:43.409388 | discovered_hf | — | — | discovery |
| 2026-11-20T21:45:43.409388 | discovered_npm | — | — | discovery |
| 2026-11-20T21:45:43.409388 | discovered_pypi | — | — | discovery |
| 2026-11-20T21:45:41.737163 | has_ci | — | False | github_api |
| 2026-11-20T21:45:41.737163 | has_docs | — | True | github_api |
| 2026-11-20T21:45:41.737163 | has_releases | — | False | github_api |
| 2026-11-20T21:45:41.737163 | has_tests | — | False | github_api |
| 2026-11-20T21:45:41.737163 | repo_languages | — | Python:77,HTML:23 | github_api |
| 2026-11-20T21:45:41.737092 | description | — | MedAgentBench: A Realistic Virtual EHR Environment to Benchmark Medical LLM Agents | github_api |
| 2026-11-20T21:45:41.737092 | license | — | MIT | github_api |
| 2026-11-20T21:45:41.737092 | repo_archived | — | False | github_api |
| 2026-11-20T21:45:41.737092 | repo_created | — | 2025-01-22T17:14:58Z | github_api |
| 2026-11-20T21:45:41.737092 | repo_forks | — | 72 | github_api |
| 2026-11-20T21:45:41.737092 | repo_homepage | — | https://ai.nejm.org/doi/full/10.1056/AIdbp2500144 | github_api |
| 2026-11-20T21:45:41.737092 | repo_last_commit | — | 2025-11-21T18:39:26Z | github_api |
| 2026-11-20T21:45:41.737092 | repo_open_issues | — | 7 | github_api |
| 2026-11-20T21:45:41.737092 | repo_primary_language | — | Python | github_api |
| 2026-11-20T21:45:41.737092 | repo_size_kb | — | 1055 | github_api |
| 2026-11-20T21:45:41.737092 | repo_stars | — | 352 | github_api |
| 2026-11-20T21:45:41.737092 | repo_topics | — | agent,benchmark,medical | github_api |