Benchmark for evaluating LLM agent ability to retrieve and answer clinical questions from FHIR-structured EHR data. Contains approximately 3,000 question-answer pairs across 100 patients with 43,000+ FHIR records spanning 15 resource types. Evaluates retrieval precision, retrieval recall, and answer correctness. Data generated by reverse-engineering and time-shifting real clinical records to reflect authentic healthcare complexity. Published at ML4H (Machine Learning for Healthcare). Developed by Verily, KAIST, and MIT.
Caution — last commit ~11 months ago; 32 stars (little traction); licence CC-BY-4.0.
Derived deterministically by the MACH scorer from the signals below — not written prose.
Maintained — last commit ~11 mo ago; has releases
★ 32 stars · 3 forks · 3 open issues
Python 99% Shell 1%
Every value records the source it was harvested from.
| Field | Value | Source |
|---|---|---|
| description | Code and Data for FHIR-AgentBench | github_api |
| discovered_conda | not_found | discovery |
| discovered_cran | not_found | discovery |
| discovered_doi | not_found | discovery |
| discovered_hf | not_found | discovery |
| discovered_npm | not_found | discovery |
| discovered_pypi | not_found | discovery |
| has CI | False | github_api |
| has docs | True | github_api |
| has releases | True | github_api |
| has tests | False | github_api |
| HuggingFace downloads (monthly) | not_found | huggingface |
| judgement | Caution | mach_scorer |
| judgement reason | Caution — last commit ~11 months ago; 32 stars (little traction); licence CC-BY-4.0. | mach_scorer |
| license | CC-BY-4.0 | github_api |
| PyPI downloads (monthly) | not_found | pypi |
| archived | False | github_api |
| repository created | 2025-09-12T05:06:56Z | github_api |
| GitHub forks | 3 | github_api |
| repo health | Maintained | mach_health |
| repo health reason | last commit ~11 mo ago; has releases | mach_health |
| homepage | https://arxiv.org/abs/2509.19319 | github_api |
| languages | Python:99,Shell:1 | github_api |
| last commit | 2025-12-15T23:10:08Z | github_api |
| open issues | 3 | github_api |
| primary language | Python | github_api |
| repo_size_kb | 3837 | github_api |
| GitHub stars | 32 | github_api |
| topics | ok | github_api |
Recorded deviations — when a harvested value changed between runs.
| When | Field | Was | Now | Source |
|---|---|---|---|---|
| 2026-11-20T21:53:45.295035 | repo_health | — | Maintained | mach_health |
| 2026-11-20T21:53:45.295035 | repo_health_reason | — | last commit ~11 mo ago; has releases | mach_health |
| 2026-11-20T21:53:45.294923 | judgement | — | Caution | mach_scorer |
| 2026-11-20T21:53:45.294923 | judgement_reason | — | Caution — last commit ~11 months ago; 32 stars (little traction); licence CC-BY-4.0. | mach_scorer |
| 2026-11-20T21:45:39.222032 | hf_downloads | — | — | huggingface |
| 2026-11-20T21:45:39.165462 | pypi_downloads | — | — | pypi |
| 2026-11-20T21:45:39.165410 | discovered_conda | — | — | discovery |
| 2026-11-20T21:45:39.165410 | discovered_cran | — | — | discovery |
| 2026-11-20T21:45:39.165410 | discovered_doi | — | — | discovery |
| 2026-11-20T21:45:39.165410 | discovered_hf | — | — | discovery |
| 2026-11-20T21:45:39.165410 | discovered_npm | — | — | discovery |
| 2026-11-20T21:45:39.165410 | discovered_pypi | — | — | discovery |
| 2026-11-20T21:45:37.506343 | has_ci | — | False | github_api |
| 2026-11-20T21:45:37.506343 | has_docs | — | True | github_api |
| 2026-11-20T21:45:37.506343 | has_releases | — | True | github_api |
| 2026-11-20T21:45:37.506343 | has_tests | — | False | github_api |
| 2026-11-20T21:45:37.506343 | repo_languages | — | Python:99,Shell:1 | github_api |
| 2026-11-20T21:45:37.506276 | description | — | Code and Data for FHIR-AgentBench | github_api |
| 2026-11-20T21:45:37.506276 | license | — | CC-BY-4.0 | github_api |
| 2026-11-20T21:45:37.506276 | repo_archived | — | False | github_api |
| 2026-11-20T21:45:37.506276 | repo_created | — | 2025-09-12T05:06:56Z | github_api |
| 2026-11-20T21:45:37.506276 | repo_forks | — | 3 | github_api |
| 2026-11-20T21:45:37.506276 | repo_homepage | — | https://arxiv.org/abs/2509.19319 | github_api |
| 2026-11-20T21:45:37.506276 | repo_last_commit | — | 2025-12-15T23:10:08Z | github_api |
| 2026-11-20T21:45:37.506276 | repo_open_issues | — | 3 | github_api |
| 2026-11-20T21:45:37.506276 | repo_primary_language | — | Python | github_api |
| 2026-11-20T21:45:37.506276 | repo_size_kb | — | 3837 | github_api |
| 2026-11-20T21:45:37.506276 | repo_stars | — | 32 | github_api |
| 2026-11-20T21:45:37.506276 | repo_topics | — | — | github_api |