← Back to catalog

CliniCARE-Bench Assess

An open clinical-agent benchmark over MIMIC-IV from Scale AI. An agent answers a clinical question about a single patient record, accesses the chart through a governed tool surface, and returns a four-way verdict (YES, NO, INDETERMINATE for lack of data, or INDETERMINATE for medical ambiguity) with cited evidence. Scoring covers verdict accuracy, evidence retrieval, citation support, process-rubric adherence, and appropriate deferral when the record cannot settle the question. Comprises 25 scenarios / 750 tasks (150 indeterminate) plus a 148-case dev subset, with an evaluation pipeline, scenario templates, and a leaderboard. The MIT licence covers the code and scenario specs; patient data is NOT distributed — the MIMIC-IV-derived cohort bundle requires PhysioNet credentials and a DUA, and corpus documents carry per-publisher terms. Early-stage: the README states it is being prepared for public release.

Judgement: Assess

Assess — committed within 3 months; 16 stars (little traction); permissive licence (MIT).

Derived deterministically by the MACH scorer from the signals below — not written prose.

Repository

Active — committed within the last month; no tests/CI/releases detected

★ 16 stars · 4 forks · 2 open issues

Python 98% Dockerfile 1% Shell 1%

Signals & lineage

Every value records the source it was harvested from.

FieldValueSource
descriptionokgithub_api
discovered_condanot_founddiscovery
discovered_crannot_founddiscovery
discovered_doinot_founddiscovery
discovered_hfnot_founddiscovery
discovered_npmnot_founddiscovery
discovered_pypiclinicarediscovery
has CIFalsegithub_api
has docsTruegithub_api
has releasesFalsegithub_api
has testsFalsegithub_api
HuggingFace downloads (monthly)not_foundhuggingface
judgementAssessmach_scorer
judgement reasonAssess — committed within 3 months; 16 stars (little traction); permissive licence (MIT).mach_scorer
licenseMITgithub_api
PyPI downloads (monthly)not_foundpypi
archivedFalsegithub_api
repository created2026-08-25T13:38:02Zgithub_api
GitHub forks4github_api
repo healthActivemach_health
repo health reasoncommitted within the last month; no tests/CI/releases detectedmach_health
homepageokgithub_api
languagesPython:98,Dockerfile:1,Shell:1github_api
last commit2026-10-26T16:48:17Zgithub_api
open issues2github_api
primary languagePythongithub_api
repo_size_kb1284github_api
GitHub stars16github_api
topicsokgithub_api

Change history

Recorded deviations — when a harvested value changed between runs.

WhenFieldWasNowSource
2026-11-20T21:53:44.641707repo_health—Activemach_health
2026-11-20T21:53:44.641707repo_health_reason—committed within the last month; no tests/CI/releases detectedmach_health
2026-11-20T21:53:44.641604judgement—Assessmach_scorer
2026-11-20T21:53:44.641604judgement_reason—Assess — committed within 3 months; 16 stars (little traction); permissive licence (MIT).mach_scorer
2026-11-20T21:52:50.294162hf_downloads——huggingface
2026-11-20T21:52:50.231597pypi_downloads——pypi
2026-11-20T21:52:50.231544discovered_conda——discovery
2026-11-20T21:52:50.231544discovered_cran——discovery
2026-11-20T21:52:50.231544discovered_doi——discovery
2026-11-20T21:52:50.231544discovered_hf——discovery
2026-11-20T21:52:50.231544discovered_npm——discovery
2026-11-20T21:52:50.231544discovered_pypi—clinicarediscovery
2026-11-20T21:52:48.492610has_ci—Falsegithub_api
2026-11-20T21:52:48.492610has_docs—Truegithub_api
2026-11-20T21:52:48.492610has_releases—Falsegithub_api
2026-11-20T21:52:48.492610has_tests—Falsegithub_api
2026-11-20T21:52:48.492610repo_languages—Python:98,Dockerfile:1,Shell:1github_api
2026-11-20T21:52:48.492537description——github_api
2026-11-20T21:52:48.492537license—MITgithub_api
2026-11-20T21:52:48.492537repo_archived—Falsegithub_api
2026-11-20T21:52:48.492537repo_created—2026-08-25T13:38:02Zgithub_api
2026-11-20T21:52:48.492537repo_forks—4github_api
2026-11-20T21:52:48.492537repo_homepage——github_api
2026-11-20T21:52:48.492537repo_last_commit—2026-10-26T16:48:17Zgithub_api
2026-11-20T21:52:48.492537repo_open_issues—2github_api
2026-11-20T21:52:48.492537repo_primary_language—Pythongithub_api
2026-11-20T21:52:48.492537repo_size_kb—1284github_api
2026-11-20T21:52:48.492537repo_stars—16github_api
2026-11-20T21:52:48.492537repo_topics——github_api