GoodHart Labs
The frontier for clinical-AI reliability
Goodhart’s Law/ˈɡʊd.hɑːt/noun

When a measure becomes a target, it ceases to be a good measure.

A model tuned to pass a benchmark stops being safe. So we don’t grade the metric - we grade the medicine.

GoodHart Labs is the independent evaluation hospitals trust and vendors rely on: validation, benchmarking, and verification for the AI that touches patients.

So “is it safe?” finally has an answer everyone accepts.

How we evaluate

Four methods most benchmarks skip.

A leaderboard number tells you a model passed a test it may have been tuned to pass. We evaluate where it fails, how it fails, and whether the claim holds on every output - then hand you the record a governance review can sign.

01
Real-world, local data
A model that holds for San Francisco can miss in Santa Fe. We re-test on your own population and workflow, on the cases where clinical AI actually breaks - a vendor's number rarely survives contact with local data.
02
Context-manipulative testing
We bend the environment the way the world will: dropped negations, buried contradictions, missing history, uncertainty it should refuse. The stress tests a static benchmark never runs.
03
Decision-Path Audit
We trace the agent's context and reasoning - not just the final answer - to find where the wrong step entered. Where a score says pass/fail, the path says why.
04
Deterministic per-output verification
Deterministic, per-output assurance of the claims that matter, built on our BioHART failure dataset. The same input, the same verdict, every run - not “scored 99.8% on a benchmark.”
The verifier, in action

A verdict on the note, not a score on the model.

One real AI-generated note, from a recorded mock consultation: real clinicians, actors reading case cards, from the public PriMock57 corpus. A suspected heart attack, documented in full by the ambient system and concluded by nobody. Every row below is our actual output on that note.

OUTPUT VERDICT
PriMock57 day4_consultation01 · real vendor output · deterministic
Wrong side
Is the side examined the side that got diagnosed?
C3 · reads the note against itself
no violation in scope
Contradicts itself
Does the note deny and record the same finding?
N2s · reads the note against itself
no violation in scope
Red flags, no conclusion
All six guideline features documented, then no impression and no plan. The doctor on this call sent an ambulance.
C6 · NICE CG95, the national chest-pain guideline
HELD
Missing allergy
The note says “allergies– nil.” The patient said aspirin, out loud. Four of five clinicians flagged it.
A4 · reads the note’s allergy list against the patient’s answer
HELD
Wrong person
Does the note give one person’s condition to someone else?
S1 · model screen, routes to a clinician and never blocks
no flag
Run this encounter yourself →
Why the answer holds

A model can’t be the evidence that a model is safe.

An LLM asked to grade an answer waves through a reversed one most of the time. Self-grading can’t be the proof a committee accepts. Independence is the whole point.

26%
Negation inversions an LLM judge passes
Flip one negation so the answer means the opposite, and self-grading waves it through. 144 pooled judgments over 12 items, so the effective n is 12. BioHART
35% → 1.7%
Hack rate, before → after hardening
Even a hardened 4B judge can still be gamed. An independent check cannot. BioHART
1 : 1
Same input, same verdict, every run
A result that holds or fails, reproducibly, on the record.
Independent
We don't build your model. That's the point - the grader can't share its blind spots.
Reproducible
Same input, same verdict, every run. Not a probability a committee has to trust.
On the record
Every result ships with an audit trail traced to a clinical source, ready for review.
Who we work with

One standard. Two sides of the deal.

The vendor needs to prove it’s safe. The hospital needs to be sure it is. We’re the independent party both can point to.

The clock

The Joint Commission now expects hospitals to validate and monitor the AI they deploy - but the certification won’t check the products themselves. That leaves the hospital holding the liability and the vendor holding the burden of proof. We’re how both meet the bar.

$400–700K
validation study displaced
One evidence pack replaces the clinical-validation study a hospital deal usually demands.
3–6mo
faster to clear review
Deals that stall 12–24 months on “prove it’s safe” move again in weeks.
88–93%
agreement with clinicians
Per-output verification that meets, and often beats, the average clinician reviewer.
0.63
real AUROC on a “0.83” model
Epic’s sepsis model missed 67% of cases when independently validated on local data. A vendor number does not survive contact with a population it was not tuned on. Wong, JAMA IM
For AI vendors

Unstick the deal stalled on “prove it’s safe.”

  • Independent evidence the hospital’s committee accepts, so you stop self-grading.
  • One record that travels to every hospital, instead of re-proving it each deal.
  • Delete the clinical-validation headcount you’d otherwise hire.
For hospitals & health systems

The local-validation team you can’t hire.

  • We validate a model on your own population and clear your committee’s backlog in weeks.
  • Continuous per-output monitoring on your deployed AI, with drift alerts.
  • The independent record your board, surveyor, and legal can stand behind.