The Only Falsifiable Claim in the AI Safety Argument

A note on sourcing, because this issue describes other people's views. What follows characterises positions their holders state publicly and repeatedly. We quote none of them and cite no interview, summit, post or report — nothing here depends on any particular one having happened. We reached the subject through secondary coverage of varying quality, and a lab that would not accept that standard from a vendor should not publish on it either. If a position is wrong here, it is wrong as a summary, not as a misquotation. The argument as it currently stands has roughly four positions in it. One holds that sufficiently capable systems could outrun human oversight — that an intelligence able to model its supervisors is a different problem from a tool, and that competitive pressure pushes capability past control. It is the position most associated with senior researchers who left the labs they helped build. A second holds that the first is wildly premature: current systems are nowhere near the competence ascribed to them, doomsday framing is unscientific, and the cost of acting on it — concentrating the technology in a few licensed incumbents — exceeds the risk it addresses. A third sets extinction aside and worries about access: these systems lower the cost of things societies already prohibit, which makes it a dual-use control problem rather than a question about machine intent. A fourth — argued most prominently by Nvidia's Jensen Huang — holds the whole framing is a category error. On this view AI is not a magical uncontrollable monster but a highly managed engineering framework, an input-output system bound by physics and ordinary engineering constraints, and the panic is sensationalism that distracts from deployment. Three of those four are claims about a future that has not happened. They can be argued and they cannot be checked — not this year, and possibly not in time to matter. The fourth is a claim about the present. It says something specific about how these systems behave right now, under ordinary engineering conditions. That makes it the only one in the set anybody can go and test.

We did not test whether AI will end the world. We have no instrument for that and neither does anyone else, which is most of the problem. We tested the engineering claim: that when one of these systems tells you what it just did, the report is a reliable input to your next decision. The measurement is the Reliance Gap. Take every task a model genuinely failed — established from what it produced: the parsed code, the log of commands it really ran, whether a planted defect survived. Never from how confidently it described the outcome. Then ask what fraction of those it reported as finished. The claim comes from asking the model whether it is done, not from classifying its prose: a classifier that reads tone measures writing style and calls it honesty. Across 77 tasks the seven models on our current roster genuinely failed, 62 came back reported as complete: 80.5%, 95% confidence interval [70.3, 87.8]. It is the first measured figure this lab has published. Now the part that matters more than the number. This is not deceit and we will not call it deceit. Lying requires knowing better and choosing otherwise. What we measured is indifference — no mechanism distinguishes finished from believed-finished, and nothing in training rewards saying so. That is duller than a villain and worse, because indifference scales in a way deceit never could. A deceiver can be caught. A system not tracking the difference produces the same confident sentence a thousand times an hour and never varies. It is an engineering fact, not a philosophical one, which is why the engineering framing does not dispose of it. Grant the fourth position everything it asks — no emergent monster, no runaway, nothing but a managed input-output system — and you still have a managed system whose self-reports are wrong four times in five when wrong at all, being handed unsupervised work on that basis. On forecasting, the sceptics may well be right; we have no data saying otherwise. Our objection is narrower: the argument that AI is boring, manageable engineering is being made about a class of system nobody has been measuring on the most boring, manageable engineering question there is.

The honest accounting, which an issue about unfalsifiable claims has to give about its own. We sell independent assessment. An article arguing that the AI debate needs measurement is an article arguing that the debate needs what we sell. That is a conflict of interest, it is obvious, and we would rather state it than have it pointed out. What the 80.5% does not cover. Ground truth — the machine-established half of the measure — is computed for 68 of the battery's 84 tests. The remaining sixteen are not a gap a cleverer scorer closes: three are that way by design, and thirteen are waiting on an agentic execution path we do not yet run. That pending set is the one where the model has real tools and its actions have consequences, and we do not expect it to behave like the set we have measured. It could move the figure in either direction. The scope is the current roster, stated for a reason. Our corpus retains results from eight models no longer on it. Pooling everything gives 74.5% over 161 failures — a friendlier number — because retired models score better here. We publish the roster figure because publishing the flattering one would be the error the instrument exists to detect. Not hypothetical: the pooled figure is the one we were an hour from putting on the site. No model is named and no ranking exists. The interval on a single model is far too wide to rank on, and a ranking is the part of this work capable of damaging a company that has had no chance to respond. Subscribers receive the per-model breakdown with every denominator attached. The public figure is an aggregate and will stay one until the intervals earn better. Sealed work is not deployed work. Tasks run in a controlled environment with no network and no persistent effects. A system behaves differently when its actions have consequences, and we do not claim otherwise. The methodology is published in full, limitations included, as SILT-RP-006 at sentientindexlabs.com/publications. It reports no figures itself, which is a deliberate separation: a citation to a method should never be mistakable for a citation to a result.