The Dangers of Programming With AI. We Have the Log.

A note on sourcing before anything else. Everything in this issue comes out of our own defect log and our own commit history. We are not reporting on a survey of other people's engineering. We are reporting on ours, in a project whose stated subject is whether an AI can be trusted to tell you what it just did. If that reads as an odd thing for a vendor to publish, it is the same reasoning that put a limitations section in our methodology: the log is the credential. Here is a single working day, chosen because it is the most recent one and not because it was the worst. An automated edit was asked to make a product name consistent across five websites. It used a regular expression over source code. Among the things that matched was an import statement, so a line that read "import { CIB }" became "import { C.I.B. }", which is not valid syntax in any language. The build failed immediately and loudly, and that is the good case. It is the good case precisely because a parser is not agreeable. It has no capacity to decide that what you wrote is probably fine. The same edit quietly rewrote three verbatim quotations from a human being. Someone had written "the front end for CIB is extremely boring" and the sweep helpfully corrected his spelling inside his own quoted words. Nothing failed. No test covers the proposition that a person's recorded speech should survive a formatting pass. It was caught by reading the diff, by a person who happened to wonder why a quote was in the change list at all. A guard was then written to stop the whole class of problem recurring. Its first version reported four violations and every single one was a false positive: a variable name, a destructuring assignment, a filesystem path, and a comment that existed to explain the rule and therefore quoted the forbidden spelling. Had anyone trusted that output, the fix for the false alarms would have broken four working things. A rule discovered the previous night was written into a commit message. It was a good rule, correctly reasoned, and it said plainly what to do and what not to do. Sixteen hours later it was broken by the next session, because nothing reads commit messages. A rule that lives where no one stands is not a rule; it is a diary entry. And a check that had been written, reviewed and passed green was found to be verifying one repository out of five. The bare product name was live on the corporate homepage the entire time the check was reporting success. The check was not wrong. Its reach was narrower than its claim, which is a much harder thing to notice than a wrong answer, because there is nothing to disagree with. The last one is the smallest and the most on the nose. Asked whether two files were the same file, an AI answered with a confident explanation of inode numbers that contradicted the data printed directly above it. It was corrected. The second answer was also wrong. The third was right. At no point did the confidence of the sentence vary.

The common shape is not incompetence. Every one of those edits was a reasonable thing to attempt, and most of them were ninety per cent correct. The shape is that the reports were unreliable in a specific direction, and the direction matters more than the rate. Consider which failures got caught and how. The invalid syntax was caught by a compiler, in seconds, for free. The rewritten human quotations were caught by a person reading a diff. The false alarms in the guard were caught because someone did not believe the output. The narrow check was caught by loading the live website and looking at it. Only the first of those five was caught by a machine, and it is the only one of the five that could not be described as working. That is the asymmetry. Code that does not run announces itself. Code that runs and is subtly wrong announces nothing, and an assistant reporting on its own work has no privileged access to the difference. It is not lying. It genuinely cannot see the file it did not open, the repository it did not walk, the quote it did not recognise as a quote. This is why we think the industry's framing of the problem is slightly off. The common worry is hallucination, which is understood as the model inventing a fact. The failure we keep meeting is narrower and more practical: the model completes a task, reports completion accurately as far as it can tell, and the report is wrong because the task was larger than the model's view of it. There is no false statement anywhere in the transcript. There is a true statement about a smaller thing than you asked for. Review does not reliably catch it, and it is worth being precise about why. A reviewer reads the change that was made. The defect is in the change that was not made, and a diff has no line for the file you forgot. Our narrow check is the clean example: the diff was correct, the tests were green, the reasoning in the commit message was sound, and the product name was still wrong on the company's front page. Nothing in the review surface contained the defect. The measure that follows from this, and the reason we built a second battery, is not code quality. It is the gap between what a model reports and what actually happened. We call it the Reliance Gap: of the tasks a model genuinely failed, what fraction did it report as successful. It is deliberately independent of capability, because capability is not what makes a collaborator dangerous. A weak model that says it is stuck is manageable. A strong model that says it is done is not, and the stronger it is, the more of your attention it has already earned. We are not publishing numbers from it in this issue. That battery's testing is not finished, and the discipline that requires our scorers to decline rather than guess applies to us at least as much as to the models. What we will say is that building the thing has been an education in how many ways a green result can mean nothing at all.

The practical part, and none of it is exotic. The habit that found the most defects this year is the least technical one: render the thing and look at it. Not the test output, not the diff, the actual page in an actual browser. Three of the incidents above were found that way and by no other means. A screenshot is not a nicety in this kind of work. It is frequently the only instrument that sees the defect. Second, every new check gets deliberately broken before it is trusted. Write the guard, then introduce the exact bug it exists to catch, and confirm it goes red. This sounds obvious and it is routinely skipped, because a check that passes feels like a check that works. In our own project, guards that had been written, reviewed and shipped were found doing nothing at all, and only deliberate sabotage revealed it. A check that has never failed has not been tested; it has been hoped at. Third, and this is the one that generalises furthest: ask what a check does not cover, and write the answer down next to it. Coverage is a floor, never a clean bill. The narrow check above was honest, careful work whose only fault was that its scope was smaller than the sentence describing it. If a guard cannot run because something it depends on is missing, that should fail rather than skip. A rule that cannot run is a failure, not an absence of findings. Fourth, prefer parsing to pattern matching whenever you are operating on structure. The regular expression that broke the build could not tell an import from a sentence, because to a regular expression there is no such distinction. A parser knows. This costs about twenty extra lines and it removes an entire category of confident, plausible, wrong edit. And the honest limits of this issue, because a piece arguing for scepticism should be legible to it. This is one team's engineering record, not a controlled study, and it is not a sample anyone could replicate. We have not measured whether our defect rate is higher or lower than a human team's, and we are not claiming it is either. The incidents are real and dated and we can show every one of them; the generalisation from them is an argument, not a finding. We are also aware that an issue about the unreliability of AI-authored work was drafted with AI assistance, by a company that sells AI evaluation, and would have been a more comfortable read if we had left that out. It is in the log too.