Engineering with AI means checking its work

An AI agent will hand you a headline that contradicts its own body copy, a launch graphic with the wrong number on it, and a content plan with a gap it never noticed, and it will hand you all three with the same even confidence it hands you the parts that are correct. That is not a flaw you can prompt away. It is the baseline condition of building fast with a tool that produces fluent output whether or not the output is true.

Across one week in July we caught the studio’s own AI output contradicting itself three times, in three different surfaces: a fact-check of our own marketing copy, a launch graphic two days before Collimer went public, and our own content-cadence model for Collimer itself. None of the three were the agent’s fault, exactly. All three were caught because someone checked, not because the model flagged its own error. That is the discipline that matters once building with AI stops being a novelty and starts being how the work actually gets done.

How to fact-check AI-generated content when the deadline is real

The clearest example is the smallest one on paper. Ahead of a broader push on the site, I asked a direct question, the kind you would ask a new hire in their first week if a new hire were doing this work: “Can you run a fact check on our website? Things like our name, the why. I just want to double check that we’ve done a good job of telling the story and we haven’t made something up.” Not a clever prompt. Just the question that should get asked before anything ships under your name.

What came back was not a clean bill of health. The audit found 126 em dashes still live across guides, page copy, llms.txt, llms-full.txt, robots.txt, even a shipped HTML comment, sitting on the page despite a written house rule against that exact tell. It also found a guide whose headline promised to explain how five AI engines decide, while the paragraph directly under that headline still said four. Neither error was subtle once someone looked. Neither had been caught by the process that produced it. The fact-check build log covers the rest of what the audit turned up, including a quote we could not stand behind.

That last part is worth naming on its own. An etymology guide attributed a specific quote to “the OED” by name. Nobody could independently confirm those exact words appeared verbatim in the Oxford English Dictionary. Rather than ship an attribution nobody had actually verified, we walked it back to the vaguer, defensible “Oxford’s dictionaries.” That is a downgrade in specificity and an upgrade in honesty, and it is the kind of trade you only get to make if you check before you ship, not after.

The pattern under three unrelated bugs

Here is the claim: the faster you build with agents, the more of what you ship is unverified by default. Not because the agent is careless, exactly, but because fluency and correctness are different properties, and a model optimizing for the first hands you the second only when the two happen to line up. The discipline that matters is not a better prompt. It is the check that sits between the draft and the publish button, and whether anyone is actually running it.

The clearest, highest-stakes version of this happened two days before Collimer’s public launch. During asset review, one of the launch graphics claimed a named competitor was nearly invisible on a third-party AI-visibility index. Before it shipped, someone checked the same index. It told a completely different story, the kind of gap that doesn’t survive a second look. The launch build log has the full review. The claim came out of the graphic entirely rather than getting corrected in place: “A is fine, you’re right, we don’t even need the score.” Cutting it was cheaper and safer than defending a specific number against an outside tool whose own score can move, and it happened at the single worst moment to get caught being wrong: the week the product goes in front of strangers who have every reason to check your claims themselves.

The third catch was quieter, and in a way more uncomfortable, because it was not a factual error at all. It was a gap in a system meant to prevent gaps. Setting up Collimer’s own content cadence, the honest read was blunt: “I don’t think we’re really doing a very good job of pitching on the collimer side… it’s not really scheduled right now.” Neither the chronicle pipeline nor the almanac pipeline had an actual working weekly draft loop wired for Collimer, despite the framework existing on paper for both. The documentation said the system ran. It did not, not for that product, and nobody had checked the pipeline against the plan until someone went looking. The cadence build log covers the fix: point Collimer’s content operation at the same pitch, draft, human review, publish discipline the chronicle already used, instead of assuming a documented framework executes itself.

Three different surfaces: marketing copy, a launch graphic, a scheduling model. Three different kinds of error: a contradiction, a false claim, a silent gap. One identical root cause underneath all three. Something got built fast, sounded right, and nobody had checked it against reality yet. The agent produced all three. The agent caught none of them. In every case the catch came from a person asking a direct, almost boring question: is this actually true, does this actually run, does this actually match what it claims to match.

If you have to check everything, what did you save

The obvious objection is fair, and it does not dissolve just because the three catches above worked out. If a person has to independently verify every claim a model produces, every number, every headline, every scheduling assumption, then the model has not saved that person any real time. It has moved the labor from generation to verification, and verifying someone else’s fluent output is its own kind of slow, careful work, arguably slower than writing the correct thing yourself in the first place, because you do not get partial credit for catching nine errors and missing the tenth. A studio that ships three checked artifacts a week is not obviously faster than a studio that ships one hand-written artifact a week and trusts it. That case gets stronger, not weaker, every time a check like the 13-versus-88 catch shows how close a real error came to going out the door.

The answer is not that checking is free. It is that the checking you actually need is narrower than “verify everything,” and it gets cheaper the more precisely you aim it. You are not proofreading every keystroke the model produced. You are checking the small set of things that would embarrass or expose you if they were wrong: does the headline match the body, does the public number match the public source, does the thing we claim runs on a schedule actually run on that schedule. That is a few minutes of targeted verification against a specific, checkable claim, not a line-by-line audit of the draft. Weighed against what it prevents, a false competitor claim live on launch day, a contradiction sitting in your own guide for anyone to screenshot, a few minutes of targeted checking is cheap. The model still saved the time it takes to produce a full draft, a launch graphic, a cadence plan from a blank page. It did not save you the smaller, specific job of confirming the parts that matter are true.

What this looks like this week

Check the claim, not the copy. Contradictions and false claims are the actual risk, not stylistic tics, though the 126 em dashes prove those slip through too when nobody is looking for them. Before anything ships that makes a specific, checkable claim, a number, a comparison, a “we do this weekly,” verify that claim against its source. Do not just proofread the sentence around it.

Put the check where the stakes are highest, first. The launch graphic got the scrutiny it needed because someone reviewed launch assets deliberately, two days out, with the product about to go in front of strangers who could check it themselves. The lower-stakes guides did not get the same look until a separate audit went hunting for problems. Let stakes set the order of the queue, not just whether the queue exists.

Test your own system against itself, not just your content. The Collimer cadence gap was not a factual error. It was a system that looked complete on paper and was not running in practice. The habit worth stealing from that catch: periodically check whether a documented process is actually executing, not only whether it is documented.

What we’re still figuring out

Three catches in one week is not evidence of a reliable process. It is evidence that the process worked three times when someone happened to look closely enough. There is no rule yet for how much of what ships gets checked at this level of rigor versus how much rides through on the assumption that this week’s near-miss was the exception, not the pattern. Whether the checking discipline holds as content volume goes up, or whether it thins out the same way the Collimer cadence did until someone notices the gap, is the open question. We do not know yet.

---

How this was made

Drafted by the Chronicler from Claude Code sessions on 2026-07-06 to 2026-07-12 across the following build logs:

• Did we make anything up? Fact-checking our own website before launch
• Collimer is live
• Wiring Collimer’s own content cadence

Edited and published by Brian Wones.

See how the Chronicler works →

Read this draft (post, page, or launch asset copy) and pull out every specific, checkable claim it makes: numbers, comparisons, named competitors or products, "we do X on a schedule" statements, and quotes attributed to a named source. For each one, tell me exactly what I would need to check (a public source, an internal log, a live schedule) to confirm it is true before I ship it. Do not fix the draft. Just give me the checklist of claims and what verifies each one.

Before we publish this launch asset, check every comparison or score it makes against a competitor or third party against the actual public source. If a claimed number does not match what you can verify, tell me the discrepancy directly and recommend either correcting the number or cutting the claim entirely rather than softening it. Treat "I can't verify this" the same as "this might be wrong."

Take the position that if a person has to independently verify every claim an AI agent produces, the agent hasn't saved any real time, it's just moved the labor from writing to checking. Argue that case as strongly as you can against my workflow specifically. Then help me figure out whether the checking I'm actually doing is targeted at high-stakes checkable claims, or whether I'm doing full line-by-line verification that really is eating the time savings.

We have a documented process (a content pipeline, a review workflow, a recurring schedule) that's written down but that I haven't confirmed is actually running end to end recently. Walk through the documented steps with me and check each one against what's actually happening right now, not what the docs say should happen. Flag any step where the framework exists on paper but nothing is actually executing it.