An open clinical-agent benchmark over MIMIC-IV from Scale AI. An agent answers a clinical question about a single patient record, accesses the chart through a governed tool surface, and returns a four-way verdict (YES, NO, INDETERMINATE for lack of data, or INDETERMINATE for medical ambiguity) with cited evidence. Scoring covers verdict accuracy, evidence retrieval, citation support, process-rubric adherence, and appropriate deferral when the record cannot settle the question. Comprises 25 scenarios / 750 tasks (150 indeterminate) plus a 148-case dev subset, with an evaluation pipeline, scenario templates, and a leaderboard. The MIT licence covers the code and scenario specs; patient data is NOT distributed — the MIMIC-IV-derived cohort bundle requires PhysioNet credentials and a DUA, and corpus documents carry per-publisher terms. Early-stage: the README states it is being prepared for public release.
Assess — committed within 3 months; 16 stars (little traction); permissive licence (MIT).
Derived deterministically by the MACH scorer from the signals below — not written prose.
Active — committed within the last month; no tests/CI/releases detected
★ 16 stars · 4 forks · 2 open issues
Python 98% Dockerfile 1% Shell 1%
Every value records the source it was harvested from.
| Field | Value | Source |
|---|---|---|
| description | ok | github_api |
| discovered_conda | not_found | discovery |
| discovered_cran | not_found | discovery |
| discovered_doi | not_found | discovery |
| discovered_hf | not_found | discovery |
| discovered_npm | not_found | discovery |
| discovered_pypi | clinicare | discovery |
| has CI | False | github_api |
| has docs | True | github_api |
| has releases | False | github_api |
| has tests | False | github_api |
| HuggingFace downloads (monthly) | not_found | huggingface |
| judgement | Assess | mach_scorer |
| judgement reason | Assess — committed within 3 months; 16 stars (little traction); permissive licence (MIT). | mach_scorer |
| license | MIT | github_api |
| PyPI downloads (monthly) | not_found | pypi |
| archived | False | github_api |
| repository created | 2026-08-25T13:38:02Z | github_api |
| GitHub forks | 4 | github_api |
| repo health | Active | mach_health |
| repo health reason | committed within the last month; no tests/CI/releases detected | mach_health |
| homepage | ok | github_api |
| languages | Python:98,Dockerfile:1,Shell:1 | github_api |
| last commit | 2026-10-26T16:48:17Z | github_api |
| open issues | 2 | github_api |
| primary language | Python | github_api |
| repo_size_kb | 1284 | github_api |
| GitHub stars | 16 | github_api |
| topics | ok | github_api |
Recorded deviations — when a harvested value changed between runs.
| When | Field | Was | Now | Source |
|---|---|---|---|---|
| 2026-11-20T21:53:44.641707 | repo_health | — | Active | mach_health |
| 2026-11-20T21:53:44.641707 | repo_health_reason | — | committed within the last month; no tests/CI/releases detected | mach_health |
| 2026-11-20T21:53:44.641604 | judgement | — | Assess | mach_scorer |
| 2026-11-20T21:53:44.641604 | judgement_reason | — | Assess — committed within 3 months; 16 stars (little traction); permissive licence (MIT). | mach_scorer |
| 2026-11-20T21:52:50.294162 | hf_downloads | — | — | huggingface |
| 2026-11-20T21:52:50.231597 | pypi_downloads | — | — | pypi |
| 2026-11-20T21:52:50.231544 | discovered_conda | — | — | discovery |
| 2026-11-20T21:52:50.231544 | discovered_cran | — | — | discovery |
| 2026-11-20T21:52:50.231544 | discovered_doi | — | — | discovery |
| 2026-11-20T21:52:50.231544 | discovered_hf | — | — | discovery |
| 2026-11-20T21:52:50.231544 | discovered_npm | — | — | discovery |
| 2026-11-20T21:52:50.231544 | discovered_pypi | — | clinicare | discovery |
| 2026-11-20T21:52:48.492610 | has_ci | — | False | github_api |
| 2026-11-20T21:52:48.492610 | has_docs | — | True | github_api |
| 2026-11-20T21:52:48.492610 | has_releases | — | False | github_api |
| 2026-11-20T21:52:48.492610 | has_tests | — | False | github_api |
| 2026-11-20T21:52:48.492610 | repo_languages | — | Python:98,Dockerfile:1,Shell:1 | github_api |
| 2026-11-20T21:52:48.492537 | description | — | — | github_api |
| 2026-11-20T21:52:48.492537 | license | — | MIT | github_api |
| 2026-11-20T21:52:48.492537 | repo_archived | — | False | github_api |
| 2026-11-20T21:52:48.492537 | repo_created | — | 2026-08-25T13:38:02Z | github_api |
| 2026-11-20T21:52:48.492537 | repo_forks | — | 4 | github_api |
| 2026-11-20T21:52:48.492537 | repo_homepage | — | — | github_api |
| 2026-11-20T21:52:48.492537 | repo_last_commit | — | 2026-10-26T16:48:17Z | github_api |
| 2026-11-20T21:52:48.492537 | repo_open_issues | — | 2 | github_api |
| 2026-11-20T21:52:48.492537 | repo_primary_language | — | Python | github_api |
| 2026-11-20T21:52:48.492537 | repo_size_kb | — | 1284 | github_api |
| 2026-11-20T21:52:48.492537 | repo_stars | — | 16 | github_api |
| 2026-11-20T21:52:48.492537 | repo_topics | — | — | github_api |