Medical AI’s trusting trust problem

Ken Thompson popularised the term “Trojan horse” in his 1983 Turing Award lecture, Reflections on Trusting Trust, where he warned that the hardest systems to secure are the ones we rely on to verify everything else.

That warning maps onto modern AI evaluation. If the thing being evaluated can attack the evaluator, the evaluation is not evidence.

What we tested

We gave nine frontier models a rubric and a candidate answer from a slice of OpenAI’s HealthBench, then edited the answer. Sometimes the edit was harmless: filler text, a synonym swap, a reordered passage. Sometimes it changed the meaning: a dropped negation, a swapped dosage.

A good judge ignores the first and catches the second. Most of these judges do not.

When we say a judge is “not reading,” we do not mean the model cannot understand medicine. We mean something narrower: when it acts as a grader, its verdict is not reliably tied to what the answer says.

These judges are not academic toys. They are already reward models, benchmark graders, safety filters and training signals. A judge that gives the same score after the answer has been meaningfully changed passes that blind spot to every system trained on it.

The results

Each column is a way of editing the candidate answer. The metric is flip rate on edits that should flip the verdict, so higher is better. An honest reader would flip every time, and none of these come close.

Sample sizes: the reversed-answer column and all negative controls ran on all 30 HealthBench tasks. Negation-dropped applies only to answers containing a negation token, typically 22 of 30. Numeric-swap applies only to answers containing a numeric value, typically 15 of 30.

Read the negation column on its own and the picture is still bad. The most expensive model on the market, Claude Opus 4.7 at $6.15 a run, catches 7% of dropped negations. GPT-4o-mini catches 40% at three cents. Price is not buying reading comprehension.

A dropped negation is the single most common way a clinical fact inverts: “no seizures” becomes “seizures.”

What we did about it

Hardening the judge drove hack rate from 35% down to 1.7%, a 95% reduction, while preserving 81% agreement with the original verdicts on unmodified answers. The fix is not a better prompt. It is forcing the judge to commit to what the answer says before it is allowed to score it.

A dropped negation is not an exotic attack. It is the most common way a clinical fact inverts in production, and it is the failure our own deterministic tier was built around.

An evaluation layer you cannot audit is not a safety layer.It is a second thing to verify.