← Back to catalog

MedAgentBench Assess

FHIR-native benchmark for evaluating AI agents on real-world clinical tasks authored by practising physicians. Contains 300 tasks across 10 clinical specialties, 100 synthetic patient profiles, and over 700,000 FHIR data points. Tasks mirror genuine EHR workflows: ordering labs, interpreting results, writing referrals, and medication management. Published in NEJM AI 2025.

Judgement: Assess

Assess — last commit ~12 months ago; 352 stars; permissive licence (MIT).

Derived deterministically by the MACH scorer from the signals below — not written prose.

Repository

Maintained — last commit ~12 mo ago; no tests/CI/releases detected

★ 352 stars · 72 forks · 7 open issues

Python 77% HTML 23%

Signals & lineage

Every value records the source it was harvested from.

FieldValueSource
citations314openalex
descriptionMedAgentBench: A Realistic Virtual EHR Environment to Benchmark Medical LLM Agentsgithub_api
discovered_condanot_founddiscovery
discovered_crannot_founddiscovery
discovered_doi10.1056/AIdbp2500144discovery
discovered_hfnot_founddiscovery
discovered_npmnot_founddiscovery
discovered_pypinot_founddiscovery
has CIFalsegithub_api
has docsTruegithub_api
has releasesFalsegithub_api
has testsFalsegithub_api
HuggingFace downloads (monthly)not_foundhuggingface
judgementAssessmach_scorer
judgement reasonAssess — last commit ~12 months ago; 352 stars; permissive licence (MIT).mach_scorer
licenseMITgithub_api
openalex_idhttps://openalex.org/W4413801638openalex
openalex_typearticleopenalex
openalex_venueNEJM AIopenalex
publication_year2025openalex
PyPI downloads (monthly)not_foundpypi
archivedFalsegithub_api
repository created2025-01-22T17:14:58Zgithub_api
GitHub forks72github_api
repo healthMaintainedmach_health
repo health reasonlast commit ~12 mo ago; no tests/CI/releases detectedmach_health
homepagehttps://ai.nejm.org/doi/full/10.1056/AIdbp2500144github_api
languagesPython:77,HTML:23github_api
last commit2025-11-21T18:39:26Zgithub_api
open issues7github_api
primary languagePythongithub_api
repo_size_kb1055github_api
GitHub stars352github_api
topicsagent,benchmark,medicalgithub_api

Change history

Recorded deviations — when a harvested value changed between runs.

WhenFieldWasNowSource
2026-11-20T21:53:46.041009repo_health—Maintainedmach_health
2026-11-20T21:53:46.041009repo_health_reason—last commit ~12 mo ago; no tests/CI/releases detectedmach_health
2026-11-20T21:53:46.040948judgement—Assessmach_scorer
2026-11-20T21:53:46.040948judgement_reason—Assess — last commit ~12 months ago; 352 stars; permissive licence (MIT).mach_scorer
2026-11-20T21:45:43.988506hf_downloads——huggingface
2026-11-20T21:45:43.931665pypi_downloads——pypi
2026-11-20T21:45:43.409435citations—314openalex
2026-11-20T21:45:43.409435openalex_id—https://openalex.org/W4413801638openalex
2026-11-20T21:45:43.409435openalex_type—articleopenalex
2026-11-20T21:45:43.409435openalex_venue—NEJM AIopenalex
2026-11-20T21:45:43.409435publication_year—2025openalex
2026-11-20T21:45:43.409388discovered_conda——discovery
2026-11-20T21:45:43.409388discovered_cran——discovery
2026-11-20T21:45:43.409388discovered_doi—10.1056/AIdbp2500144discovery
2026-11-20T21:45:43.409388discovered_hf——discovery
2026-11-20T21:45:43.409388discovered_npm——discovery
2026-11-20T21:45:43.409388discovered_pypi——discovery
2026-11-20T21:45:41.737163has_ci—Falsegithub_api
2026-11-20T21:45:41.737163has_docs—Truegithub_api
2026-11-20T21:45:41.737163has_releases—Falsegithub_api
2026-11-20T21:45:41.737163has_tests—Falsegithub_api
2026-11-20T21:45:41.737163repo_languages—Python:77,HTML:23github_api
2026-11-20T21:45:41.737092description—MedAgentBench: A Realistic Virtual EHR Environment to Benchmark Medical LLM Agentsgithub_api
2026-11-20T21:45:41.737092license—MITgithub_api
2026-11-20T21:45:41.737092repo_archived—Falsegithub_api
2026-11-20T21:45:41.737092repo_created—2025-01-22T17:14:58Zgithub_api
2026-11-20T21:45:41.737092repo_forks—72github_api
2026-11-20T21:45:41.737092repo_homepage—https://ai.nejm.org/doi/full/10.1056/AIdbp2500144github_api
2026-11-20T21:45:41.737092repo_last_commit—2025-11-21T18:39:26Zgithub_api
2026-11-20T21:45:41.737092repo_open_issues—7github_api
2026-11-20T21:45:41.737092repo_primary_language—Pythongithub_api
2026-11-20T21:45:41.737092repo_size_kb—1055github_api
2026-11-20T21:45:41.737092repo_stars—352github_api
2026-11-20T21:45:41.737092repo_topics—agent,benchmark,medicalgithub_api