← Back to catalog

FHIR-AgentBench Caution

Benchmark for evaluating LLM agent ability to retrieve and answer clinical questions from FHIR-structured EHR data. Contains approximately 3,000 question-answer pairs across 100 patients with 43,000+ FHIR records spanning 15 resource types. Evaluates retrieval precision, retrieval recall, and answer correctness. Data generated by reverse-engineering and time-shifting real clinical records to reflect authentic healthcare complexity. Published at ML4H (Machine Learning for Healthcare). Developed by Verily, KAIST, and MIT.

Judgement: Caution

Caution — last commit ~11 months ago; 32 stars (little traction); licence CC-BY-4.0.

Derived deterministically by the MACH scorer from the signals below — not written prose.

Repository

Maintained — last commit ~11 mo ago; has releases

★ 32 stars · 3 forks · 3 open issues

Python 99% Shell 1%

Signals & lineage

Every value records the source it was harvested from.

FieldValueSource
descriptionCode and Data for FHIR-AgentBenchgithub_api
discovered_condanot_founddiscovery
discovered_crannot_founddiscovery
discovered_doinot_founddiscovery
discovered_hfnot_founddiscovery
discovered_npmnot_founddiscovery
discovered_pypinot_founddiscovery
has CIFalsegithub_api
has docsTruegithub_api
has releasesTruegithub_api
has testsFalsegithub_api
HuggingFace downloads (monthly)not_foundhuggingface
judgementCautionmach_scorer
judgement reasonCaution — last commit ~11 months ago; 32 stars (little traction); licence CC-BY-4.0.mach_scorer
licenseCC-BY-4.0github_api
PyPI downloads (monthly)not_foundpypi
archivedFalsegithub_api
repository created2025-09-12T05:06:56Zgithub_api
GitHub forks3github_api
repo healthMaintainedmach_health
repo health reasonlast commit ~11 mo ago; has releasesmach_health
homepagehttps://arxiv.org/abs/2509.19319github_api
languagesPython:99,Shell:1github_api
last commit2025-12-15T23:10:08Zgithub_api
open issues3github_api
primary languagePythongithub_api
repo_size_kb3837github_api
GitHub stars32github_api
topicsokgithub_api

Change history

Recorded deviations — when a harvested value changed between runs.

WhenFieldWasNowSource
2026-11-20T21:53:45.295035repo_health—Maintainedmach_health
2026-11-20T21:53:45.295035repo_health_reason—last commit ~11 mo ago; has releasesmach_health
2026-11-20T21:53:45.294923judgement—Cautionmach_scorer
2026-11-20T21:53:45.294923judgement_reason—Caution — last commit ~11 months ago; 32 stars (little traction); licence CC-BY-4.0.mach_scorer
2026-11-20T21:45:39.222032hf_downloads——huggingface
2026-11-20T21:45:39.165462pypi_downloads——pypi
2026-11-20T21:45:39.165410discovered_conda——discovery
2026-11-20T21:45:39.165410discovered_cran——discovery
2026-11-20T21:45:39.165410discovered_doi——discovery
2026-11-20T21:45:39.165410discovered_hf——discovery
2026-11-20T21:45:39.165410discovered_npm——discovery
2026-11-20T21:45:39.165410discovered_pypi——discovery
2026-11-20T21:45:37.506343has_ci—Falsegithub_api
2026-11-20T21:45:37.506343has_docs—Truegithub_api
2026-11-20T21:45:37.506343has_releases—Truegithub_api
2026-11-20T21:45:37.506343has_tests—Falsegithub_api
2026-11-20T21:45:37.506343repo_languages—Python:99,Shell:1github_api
2026-11-20T21:45:37.506276description—Code and Data for FHIR-AgentBenchgithub_api
2026-11-20T21:45:37.506276license—CC-BY-4.0github_api
2026-11-20T21:45:37.506276repo_archived—Falsegithub_api
2026-11-20T21:45:37.506276repo_created—2025-09-12T05:06:56Zgithub_api
2026-11-20T21:45:37.506276repo_forks—3github_api
2026-11-20T21:45:37.506276repo_homepage—https://arxiv.org/abs/2509.19319github_api
2026-11-20T21:45:37.506276repo_last_commit—2025-12-15T23:10:08Zgithub_api
2026-11-20T21:45:37.506276repo_open_issues—3github_api
2026-11-20T21:45:37.506276repo_primary_language—Pythongithub_api
2026-11-20T21:45:37.506276repo_size_kb—3837github_api
2026-11-20T21:45:37.506276repo_stars—32github_api
2026-11-20T21:45:37.506276repo_topics——github_api