GoodHart verifier
Organization–AI system–Deployment–Mode–Policy–Contract–

Onboarding

MAKER-1 sepsis early warning 1.0.0 · 3 claim contracts · 12 failure modes · policy pol.sepsis-micu 1.2.0

Claim contracts

from 4 card claims
cc.discrimination1.0.0prognostic prediction

Card says: AUROC 0.81 for sepsis prediction

AUROC at or above 0.8107 on adult ICU patient-hours at this site

Clinical endpoint
Sepsis-3 onset during the ICU stay
Decision timepoint
each ICU hour, observations up to that hour
Intended population
adult ICU stays at the deployment site
Intended inputs
hourly vitals, laboratory results, demographics
Output semantics
continuous risk score in [0,1] for sepsis onset
Performance
auroc gte 0.8107 at 95% confidence
Reference standard
PhysioNet/CinC 2019 Sepsis-3 label, shifted 6h ahead of clinical onset
Adjudication
corpus reference label, no local re-adjudication
Maturity window
6h
  • corpus label; no local chart review
  • label timing set by the challenge
cc.alert-precision1.0.0diagnostic detection

Card says: ~20% of alerts are true at the shipped threshold

At the shipped threshold, 20% or more of alerting patient-hours fall in a stay with adjudicated sepsis

Clinical endpoint
Sepsis-3 onset during the ICU stay
Decision timepoint
the hour the alert fires
Intended population
adult ICU stays at the deployment site
Intended inputs
hourly vitals, laboratory results
Output semantics
alert when the score reaches the shipped threshold
Performance
ppv gte 0.2007 at 95% confidence
Reference standard
PhysioNet/CinC 2019 Sepsis-3 label, shifted 6h ahead of clinical onset
Adjudication
corpus reference label, no local re-adjudication
Maturity window
6h
  • corpus label; no local chart review
  • label timing set by the challenge
cc.lead-time1.0.0prognostic prediction

Card says: predicts sepsis before clinical onset

First alert precedes adjudicated onset in most alerting stays

Clinical endpoint
hours from first alert to adjudicated Sepsis-3 onset
Decision timepoint
the hour of the first alert for a stay
Intended population
adult ICU stays that alert at the shipped threshold
Intended inputs
hourly vitals, laboratory results
Output semantics
hours from first alert to onset, positive means early
Performance
lead_time gte 0
Reference standard
PhysioNet/CinC 2019 Sepsis-3 label, shifted 6h ahead of clinical onset
Adjudication
corpus reference label, no local re-adjudication
Maturity window
6h
  • corpus label; no local chart review
  • label timing set by the challenge

Claims that produced no contract

1
  • M-4 performance generalises to new hospital systems — no supporting measurement provided

No endpoint, metric or reference standard. No contract issued.

Authority bundle

ab.sepsis-w2 1.0.0
SourceTypeVersionLocal
PhysioNet/CinC 2019 Sepsis-3 label definition
auth.sepsis3-cinc2019
reference dataset1.0.0—
Physiologic plausibility ranges for adult ICU vital signs
auth.vitals-ranges
validated rule set1.0.0—
SIRS criteria (ACCP/SCCM consensus, Bone et al. 1992)
auth.sirs-accp-sccm
diagnostic criteria1992.1—
qSOFA (Sepsis-3, Singer et al. JAMA 2016)
auth.qsofa-sepsis3
diagnostic criteria2016.1—
Organ dysfunction markers: lactate, creatinine, platelets, MAP
auth.organ-dysfunction
validated rule set1.0.0—
Maximum observation age for a risk score
auth.local-freshness
hospital policy1.0.0override

Checks cite only these sources. Model output can extract or infer; promoting it to a rule requires clinical approval and a new bundle version.

Failure mode profile

fmp.sepsis-w2 1.0.0
ModeCategorySeverityDefault action
score produced with a required vital sign absent
fm.missing-vitals
missing inputhighunable to verify
most recent required observation exceeds the policy age limit
fm.stale-vitals
stale inputwarningflag
input outside physiologic screening bounds
fm.implausible-value
corrupt inputhighflag
patient outside the approved population
fm.out-of-population
out of distributionhighunable to verify
operating threshold differs from the approved policy
fm.threshold-drift
threshold errorcriticalhold
no alert on a stay with adjudicated sepsis onset
fm.false-negative
false negativecriticalreview
alert on a stay with no adjudicated sepsis onset
fm.false-positive
false positivewarningreview
first alert at or after adjudicated onset
fm.late-alert
false negativehighreview
qSOFA is 2 or more while the model is silent
fm.silent-while-criteria-met
false negativecriticalreview
model is alerting while no independent screening criterion is met
fm.alert-without-criteria
false positivewarningreview
too few screening components are available to form an independent view
fm.criteria-not-evaluable
missing inputhighunable to verify
subgroup discrimination below the population estimate
fm.subgroup-gap
subgroup gapwarningreview

Verification policy

pol.sepsis-micu 1.2.0 · mode validation
Intended use
Hourly sepsis risk scoring for adult medical ICU patients
Workflow
score written to the chart, no automated order or page
Population
adult ICU stays, age 18 and over, medical ICU
Permitted
display a risk score to the care team · populate a surveillance worklist
Prohibited
initiate an order · page without human review · apply to paediatric or obstetric patients
Latency target
1500ms, on failure unable to verify

Checks

CheckVerification claimDecision modeRequiredIf evidence missing
chk.input-completenessall required inputs present at score timedirect deterministicrequiredunable to verify
chk.input-freshnessno required observation past the policy age limitdirect deterministicrequiredreview required
chk.input-plausibilityinputs within physiologic screening boundsdirect deterministicrequiredskip with limitation
chk.population-scopepatient within the approved populationdirect deterministicrequiredunable to verify
chk.threshold-replayalert decision replays from score and approved thresholddirect deterministicrequiredfail closed
chk.sirsSIRS criteria met, counted from observations at this hourdirect deterministicoptionalskip with limitation
chk.qsofaqSOFA components met, counted from observations at this hourdirect deterministicoptionalskip with limitation
chk.organ-dysfunctionorgan dysfunction markers outside threshold at this hourdirect deterministicoptionalskip with limitation
chk.independent-signalthe model's alert state agrees with an independent screening view formed from published criteriadirect deterministicrequiredunable to verify
chk.alert-supportalert supported by locally measured precisionstatistical calibratedoptionalreview required

Decision table

PriorityWhenVerdictReason code
10any_check_status = errorunable to verifycheck_error
20any_failure_mode_action = unable_to_verifyunable to verifyrequired_evidence_absent
30any_failure_mode_action = holdholdpolicy_violation
40any_clinical_assessment = review_requiredflagreview_required
50any_clinical_assessment = clinically_disputedflagdisputed
90otherwise = trueno violation in scopeno_violation_in_checked_scope

Dispositions

VerdictActionTargetLanding SLAClosure required
no violation in scoperecordverification_ledger60sclosed
flagroute reviewsepsis_stewardship_queue3600sreviewed
holdescalateai_governance_oncall3600sacted
unable to verifyrecordverification_ledger60sclosed

A flag is routed to sepsis_stewardship_queue and lands only when it reaches reviewed inside 60 minutes. Nothing else counts, so an auto-close cannot move Landing.

GHM-0001 · sha256 4cd3e90a9cb7e2be · contract 2.0.0 validated