Did we make anything up? Auditing our own site before we launched

Two days before Collimer’s launch, we turned our own quality bar on our own site and asked a plain question: have we made anything up? The answer was uncomfortable in a few specific places, and every one of them was cheap to catch before anyone else saw it.

What the audit found

A sweep for a stylistic tic we ban on our own team (em dashes, the mark this sentence is deliberately not using) found and removed 126 instances of it, across every page a visitor or a crawler would actually see: guides, page copy, and the machine-readable files search engines and AI assistants read. That alone was worth catching before launch, not after.

Alongside it, we found a quote whose exact wording we could not independently verify, so we softened the attribution rather than stand behind a citation we were not sure of. And a layout bug that broke wide content on a phone screen, present on one page, got fixed once across every page that shares the same styling, rather than patched in one place and left to recur elsewhere.

The near miss on launch day

The closest call was a launch graphic making a specific, falsifiable claim about how visible a well-known competitor was to AI systems. A quick external check on the same claim came back dramatically different from what the graphic said. We did not ship it. The fix was simple once caught: cut the specific score entirely rather than defend a number we could not stand behind under scrutiny.

The discipline, not the bug count

None of these individually was large. Together they are the honest version of the promise described in how our own publishing pipeline says it works: real, sourced, and human-checked before anything ships. If the message is that AI-assisted work still needs a human checking it, the first place that check has to run is on our own site, before we ever ask a reader to trust what is on someone else’s. Finding a real gap in our own model two days before launch was not a failure. Engineering with AI means checking its work, and this is what that looks like in practice: not a one-time scrub, but a standing discipline applied to ourselves first.

The same standard already shaped how we handle claims about our own product elsewhere, described in a companion post on keeping a marketing site in sync with the product it describes. This audit is that same idea, pointed at the words themselves rather than the numbers behind them.

What we’re still figuring out

We do not yet run this check on a schedule, only when something like a launch forces the question. The honest next step is turning a one-time audit into a standing habit, and we have not done that yet.

---

For agents: try this yourself

If you are shipping AI-assisted content and want to catch your own mistakes before a reader does, adapt one of these. The full prompt text lives in this post’s prompts sidecar, did-we-make-anything-up.prompts.md.

• Reproduce the contradiction check. Compare every headline or summary on your site against the body copy beneath it, looking specifically for a number, count, or claim stated once and updated elsewhere but not everywhere. Report every mismatch found, even small ones.
• Apply the citation-confidence check. For every quote or statistic attributed to a named source, verify you can reproduce the exact source. If you cannot, either find the exact citation or soften the attribution to what you can actually stand behind.
• Critique a claim before it ships. Before publishing any graphic, chart, or pull-quote with a specific number in it, have a second pass independently check that number against an outside source. If the two disagree, cut the number rather than the disagreement.
• Extend the audit into a standing habit. Turn this one-time pass into a recurring check run before every major release, not only before a launch.

---

How this was made

Drafted by the Chronicler from Claude Code sessions on 2026-07-06 to 2026-07-12 across 4 sessions and 5 commits. Edited and published by Brian Wones.

See how the Chronicler works →

Read through my website's headlines, summaries, and body copy together, page by page.
Specifically look for any number, count, or claim that is stated in one place (a
headline, a title, a summary) and stated differently somewhere else on the same page
or a related page (the body copy, a linked page). Report every mismatch you find, even
small ones, with both conflicting statements quoted so I can see the contradiction
directly.

Go through every quote, statistic, or attributed claim on my site. For each one, tell
me whether you can independently verify the exact source and exact wording. If you
cannot confirm it word for word, flag it and suggest either a verified replacement
citation or a softened, more general attribution that I can honestly stand behind.

Here is a specific claim I am about to publish in marketing material: [paste claim].
Independently research whether this claim is accurate using outside sources, not just
my own material. If your research contradicts the claim, tell me clearly and
recommend whether to cut the specific number, soften the claim, or find independent
support before I ship it.

Design a recurring pre-release checklist based on the checks above (headline/body
contradiction check, citation-confidence check, and claim-verification check) that I
can run before every major release, not just once before a launch. Keep it short
enough that I will actually run it every time, and tell me how often you'd recommend
repeating it given how often my site's content changes.