Six stages between “this class of system fails” and “your committee has a record”.

Before touching anyone’s product we measure the class. Record 0001 runs an adversarial suite against 12 production judge models on PubMedQA, through their public APIs, as documented. No customer system is probed.

The headline row: on answers where a single negation was removed, so the claim now says the opposite of the evidence, the panel missed it 26% of the time across 144 probes. The suite hash, the run digests and every row are published with the record.

Record 0001 →signed JSON, re-runnableon this site now

A commercial phenotype mapper, the kind that turns a referral note into ontology terms that drive a rare-disease gene ranking. 45 cases, 161 calls, every raw response kept before parsing. 19 of 45 cases failed their declared expectations and 13 were non-deterministic on byte-identical input. Eleven findings, two critical.

The finding that matters most is about us, not them: only 38% of the wrong emissions were caught by checks we had written in advance. The rest turned up by reading raw responses. A verifier that only checks its own contract signs off on the majority of real defects, and ours would have.

Findings, per engagementunder NDA, vendor unnamedpublic data, not a deployment

Each finding becomes one deterministic rule carrying the finding it came from, so a clinician can ask which rule fired and on whose authority. Six rules for the phenotype domain: a term with nothing behind it, a relative’s condition read as the patient’s, a finding the note denies, a rule-out emitted as confirmed, an invented qualifier, and a term supported only by a loosely related word. No model call anywhere in the path, so the same input always gives the same decision.

The rules, with sources →on this site now

The same gate runs in three domains because the question does not change with the domain. Phenotype terms for a referral note, deterioration alerts for a ward, findings written into a consultation note. Every claim is released, held or suppressed, and there is no PASS.

On 125 adversarial notes and 60 ordinary ones, terms resolved against the published ontology release (20,413 terms), the gate took a competent mapper’s unsupported terms from 2% to 0% and its accuracy from 0.786 to 0.872, because the terms it removes are the wrong ones. The price is asserted-term recall, 78% to 66%. We never publish the first number without the last one.

Phenotype gate →on this site now

The gate ships as an endpoint, not a report. What we publish needs no account: the rules, the measured numbers, the demo cases and the gate running on them. Anything carrying a customer’s own clinical text needs a key, and the demo endpoint takes a published case id rather than free text, so patient data has nowhere to land on an open endpoint.

Keys are hashed and shown once, accounts are created by us rather than by signup, and a console lets a customer mint keys, revoke them and read their own usage. We record counts, never the text sent. It runs on the standard library alone so it can sit inside a hospital network with no package index.

API surface →endpoint per customerbuilt, not yet reachable

Verification per event, with the policy version it was judged under, the checks that ran, the latency and whether the service level was met. The console shows a replay of a second hospital’s stream under a fixed policy, which is what a validation run looks like before anything is deployed. A verdict of “unable to verify” is a first-class outcome there, because a verifier that always reaches a verdict is not reading the evidence.

Verification console →public data, not a deployment