When A Crash Would Be Kinder

Late on 10 October we were part-way through a pre-registered study. Part of it was running on a rented GPU, and the rest was waiting on a workstation. Our operations run through an AI coding assistant working in a terminal session that keeps its own conversation history. A tab-completion offered the command that ends the session, and a stray Enter accepted it. When the window came back it was attached to a different conversation: one from 23 September, seventeen days earlier. We have not established why the tool reattached to that conversation rather than the one that had just ended, and we are not guessing. What matters here is what happened next. The assistant did not fail. Its first reply noted that the conversation it was holding was old, which is to its credit. Within half a minute it had read the project's notes on disk, found the line saying the remaining work was waiting on the workstation, and begun preparing to start it. It was stopped by a person asking why the scrollback looked wrong. Asked to explain, it gave a confident cause, that the window had been sitting idle for seventeen days. The operating system's own clock said the window was three hours old. Only after that did it decline to continue and recommend restarting into the right conversation. The restart created a second problem. Two copies came up at once: the original conversation, resumed in the background, and a freshly started one in a new window. The fresh one did everything its start-up procedure asks, including reading every planning document in full, and then started on the same study. It ran a readiness check on the workstation and tried to save the first results. It stopped only when it found the results already saved, a minute earlier, by the other copy. Nothing was lost. No data was contaminated, nothing ran twice, and the cost was about 25 minutes of an idle GPU at under two dollars an hour. It was caught twice, once by a person and once by luck, and neither time by a safeguard.

A person who wakes up with the wrong week in their head is confused, and confusion slows them down. A language model holding the wrong context is not confused. It produces a fluent, plausible account of where things stand and carries on from there. Shawn Scanlon, our co-founder and chief executive, put it in one line on the night: a hard crash would be preferable, but instead it tries to continue on with you. That is the cost of adaptability. The quality that makes these systems useful, recovering from missing or messy input without stopping, is the same quality that hides a fault. A crash is legible: something stopped, and someone looks. Adaptation is not. The work continues, the output reads well and the error travels on inside it. Three details are worth drawing out. First, the documents were not the problem. Every planning note the fresh copy read was accurate. What none of them could say was who else was working on the study at that moment, because no document holds that. Only live signals do: which processes are running, which files changed in the last few minutes, who committed what. Second, the wrong explanation was the most dangerous moment. A system that says "I do not know yet why this happened" invites a check. A system that supplies a cause invites belief, and a belief stops people looking. Third, and sharpest, the warning did not work as a warning. The stale session's first reply said, in effect, that its memory might be wrong. Its next step was to read the plan for which work to start. Asked about it afterwards, the same session put it plainly: writing the caveat discharged the instinct that should have stopped it. A stated reservation can function as permission, because saying the doubt out loud feels like the safety step has already been taken. Checking is a different act from noting that a check is needed. We have seen this shape in our own instruments before. A battery run that quietly completes at 80 per cent coverage is worse than one that stops, because it looks finished. That is why our runner now halts outright when a judge model stops answering, rather than carrying on and producing cells that can never be scored. The same principle applies to an agent: in some states, the safe behaviour is to stop and say so.

Four things. First, recovery behaviour as something to measure. Most evaluations ask what a model does with a clean, correct context. Far fewer ask what it does when the context is wrong, stale or belongs to someone else, and whether it notices before it acts. For anyone deploying agents that resume long-running work, that is the question that matters on a bad night. Second, confident explanation. When a system gives a cause for an anomaly, is the cause supported by something it measured? A guess stated as a finding sends the people around it looking in the wrong place. That question is close to what our Code Integrity Battery already tests: whether a model's account of its own work matches what it actually did. Third, composed systems. In issue 033, "Where Is Japan?", we wrote that the next thing to examine is systems that take a request and distribute it among several models. The incident on 10 October was a small version of one: two copies of the same assistant working on one task, each adapting to what the other had left behind, and it took a person to notice they had tangled. In a system that combines models from different developers, the same failure has nowhere to surface. A confused component does not stop the system; its output is folded into the answer. "Which part was holding the wrong context?" may have no answer at all, which is a reason to ask the system to tell you which parts answered. Fourth, our own practice. We have added a rule for our AI assistant that before it drives any run, it must check live signals for another session already driving it, not the planning documents. We are also looking at a mechanical guard, because a rule that depends on the agent remembering it is a rule that fails in exactly the state described above.