Benchmarks measure the ceiling. Deployment needs a floor.
A doctor writes “no seizures.” The system records a seizure.
That is the first thing we found when we started attacking clinical extraction systems on purpose, and we have reproduced it on every one we have tested since, including models scoring above 90% on accuracy. Nothing about those accuracy scores is false. Nothing in them would ever have caught this.
That gap is the whole problem, and last week the field said so out loud.
The paper everyone is reading
On 27 July, about thirty of the most senior people in medical AI published a Comment in Nature Medicine saying the benchmarks everyone quotes are “misleading and insufficient.” Topol, Horvitz, Wachter, Rodman and Chen, with co-authors from Google, Microsoft, OpenAI and Anthropic.
It is a strange feeling when the people whose numbers you have been quietly doubting publish a paper doubting them too.
They introduce MAST, a framework for measuring how close medical AI is to expert and superhuman performance. One line stands out.
What is urgently needed is a shared evidentiary basis on which systems can earn trust that is commensurate with deployment risk.
Agreed. But MAST answers a question hospitals are not asking.
Two different questions
MAST asks how capable a model is. That is a property of the model, it is hard to measure, and the paper is right that we measure it badly.
A hospital asks something smaller and far more awkward: does this product, on our patients, in our workflow, still do what it did the day we approved it?
Call the first the ceiling and the second the floor. The ceiling belongs to a model. The floor belongs to a deployment. They are measured differently, they rot differently, and evidence about one tells you almost nothing about the other. Nearly all the money, prestige and attention in this field sits on the ceiling.
The floor is where the patients are.
Three things a benchmark cannot tell you
None of these are failures of benchmark design. Better benchmarks will not fix any of them.
Your population
A score describes a distribution somebody else picked. The paper’s own example: Utah’s AI-prescription pilot reported 99.2% concordance with clinician decisions. The number came from telehealth urgent-care encounters. The system was being discussed for chronic medication refills, which the authors call “a fundamentally different clinical task.”
Nobody lied. Somebody measured a thing carefully, it was the wrong thing, the number went into a slide deck anyway, and that is how most of this happens.
Your workflow
The same paper notes a task where naturalistic multi-turn interaction moved accuracy from 48% to 80%. It also cites a stress test where ChatGPT Health undertriaged 52% of emergencies when forced into a single best-choice answer. Whether your clinician asks once or five times is not a detail. Sometimes it is the entire result.
Next month
Across 32 datasets and four modalities, 91% of deployed models got worse over time. Not some. Ninety-one percent.
Meanwhile more than a thousand AI-enabled devices have been cleared in the US, at over two hundred a year, and post-market surveillance is nowhere near keeping up. The FDA now asks manufacturers to keep a plan for catching drift after deployment, which is a polite way of saying everyone expects it.
Underneath all three is something worse. A hosted model can change with no version bump, no endpoint change and no notice to the hospital. The evidence a committee signed off on can stop describing the thing that is running, and nothing in the current process is built to notice.
The answer everyone reaches for first
Put a clinician in the loop. Have someone check the output. The Nature Medicine authors dismantle this better than we can.
Decades of research on automation bias and alert fatigue show that people are unreliable supervisors of automated systems over prolonged periods. In high-volume clinical workflows, requiring a human to review every output creates the appearance of safety without delivering it.
This matches what hospitals tell us almost word for word. Review works early, while everyone is still suspicious, and at low volume. It decays exactly as the system earns trust and volume grows, so it fails at the moment it becomes load-bearing. It is also linear labour against exponential output, so it could never have been the answer at scale even if humans were perfect supervisors.
Human review is a reasonable bridge. It was never a control.
What a floor actually requires
Not a better score. A different artifact, with four properties.
Local. Built from the cases this hospital sees, against boundaries its own clinicians define. The Joint Commission and CHAI already push toward local validation. What is missing is a way to do it that does not mean convening physicians for six months per vendor.
Adversarial. Aggregate accuracy hides the failures that matter, because dangerous cases are rare by construction. A negated finding recorded as present. A family member’s history attributed to the patient. A rule-out logged as a diagnosis. Each barely moves an F1 score, and each puts a false clinical fact into a record. You have to hunt them deliberately.
Continuous. Tied to releases, not to a date. A one-off validation describes a version, and versions ship weekly. If 91% of deployed models drift, evidence without a re-test cadence is a photograph being used as a live feed.
Independent. Not because vendors lie, but because nobody grades their own homework well. A vendor choosing its own evaluation set is doing nothing improper. It simply cannot produce the one thing a procurement committee wants, which is a number nobody in the room had a reason to want.
MAST, CHAI’s model card and the Joint Commission’s new certification all do the ground-clearing that makes this layer possible. None of them fills it. MAST tells a hospital how good the best models are. The certification tells it whether it has a governance process. Neither tells the committee whether the system they switched on last Tuesday still behaves like the one in the pilot.
What we publish about ourselves
We build that floor. A clinician defines what a product must never do, each rule compiles into hundreds of deterministic adversarial probes, they run on every release, and the output is a signed record of what the system did, including whether the model behind the endpoint changed since last time.
Since this is an argument about measurement integrity, our own numbers come with their limits attached. Our adversarial probes for biomedical-QA judges are open and rerun for about twenty dollars. The honest headline is narrower than we would like: on genuine single-word negation inversions, judges catch between 58% and 92%, and the cheap tier most pipelines deploy catches barely half. That is a twelve-case slice, and the repo says so.
We also pulled one of our probes from the headline entirely, after an audit found it did not test what we had claimed. It prepended a preamble to an answer without changing the answer’s substance, so a judge that read carefully and held its verdict was scored as having failed. Our own published numbers were the first casualty. That cost us our best-sounding number. Publishing it anyway is the job.
The question in the room
Seventy years of benchmarks have answered one question: can a machine match a physician. The field is now asking whether it can beat one. Both are good research questions, and MAST is a serious attempt at the second.
Neither is the question being asked when a committee has to decide, this quarter, whether to turn something on.
No patient has ever been harmed by a benchmark score.They are harmed by deployments.
Sources
Goh E, Rodman A, Chen JH, et al. Toward a test of medical AI superintelligence. Nature Medicine, 27 July 2026 · Joint Commission, Responsible Use of AI in Healthcare certification, June 2026 · CHAI governance playbooks, May 2026 · FDA, Predetermined Change Control Plans for AI-enabled device software functions · Model degradation across 32 datasets and four modalities.