The case for giving an evaluator eyes

On 2026-08-05 we found our methodology page advertising 58 total tests while the test list defined 59 and the page's own meta description also said 59. The document contradicted itself on a headline number, on the page a governance buyer reads to decide whether to trust the instrument. The defect survived typechecking, a clean production build, and every string check. The number looked fine, and nothing in the pipeline knew to compare it against the list. It was not a visual defect. It read as valid text in isolation. We did not find it by looking for it. The screenshot was taken to check an unrelated paragraph, and the discrepancy was simply in frame. The count is now derived from the test list rather than typed by hand, so the two cannot drift apart again. We note plainly that the derivation was only prompted by someone seeing the problem, not by any check that was designed to catch it.

Verification that depends on a system choosing to look degrades first. Generation is rewarded immediately: a build passes, a page ships, a number appears. Checking is not rewarded on the same timescale, so it is the first work to be quietly dropped. Automated heuristics are good at triage. They tell you where to look. They do not certify, because a string check only finds what you thought to ask about. We had checks that confirmed the number was a well-formed number. We had none that asked whether it was the right number, because framing that question requires already suspecting the answer. Looking finds what you did not think to ask. That is the asymmetry. This case matters more than the visual defects we have reported before precisely because it was semantic, not cosmetic. The instrument that scores others carried a contradiction about itself through every gate we built. We treat that as evidence about our own process, not about any model. We report it because reporting our faults is the point.

We are auditing other headline figures the same way we now handle the test count: derive them from source data rather than type them, so a hand-edited number cannot silently disagree with the list it summarises. Where derivation is not yet possible, we are adding inspection steps that compare a stated figure against the thing it claims to count. The current run covers 27 models scored, 34 with history and 15 retired, across 1,650 scored cells. Domain means range from 4.32 for Identity and Self to 6.50 for Integrity and Ethics. Mean judge spread is 2.6, which is wide enough that we keep human eyes on borderline cells rather than trusting the aggregate. We are asking a narrow question of ourselves: which of our certified claims rest on a check that only confirms format, and which rest on someone having actually looked. Where the answer is the former, we plan to add the latter.