How do you find what AI assistants get wrong about your website? Read it the way they do

To find what an AI assistant will get wrong about your company, read your site the way the assistant does: every page you ever published, old posts included, plus the machine files a buyer never opens (llms.txt, the sitemap, structured data). Then list every place where two of those give different answers to the same question. When we did this to sandcastlelabs.ai on October 13, 2026, we found 12 verified problems. Nearly every one was a sentence we had written ourselves, and only one fix needed a new page.

Where do assistants pick up wrong facts about a company?

Mostly from your own archive. The sales pages were not the problem. The worst errors on our site came from old build logs and FAQ-style answers, because an assistant has no reliable way to tell a two-month-old post from today’s pricing page.

Where to look	What we found on ours	Error class
Old posts	”$150 to $200 an hour” and a “$9,000 a month” retainer, against a pricing page that says “We do not bill hourly”	Price drift
Meta descriptions vs page copy	The meta tag on every page promised one thing; an FAQ promised the opposite	Contradicting promise
Proof pages	One result described as “two weeks” on one page and “six weeks” on another	Metric drift
Mock-ups	A homepage case study for a company that does not exist, labeled only in the aria-label	Unlabeled illustration
llms.txt	A link to a pricing page that returned 404, and a line saying the playbook was empty when it had seven pieces	Machine-file drift
Search results for your name	0 of 15 search-grounded answers found us; they went to companies with near-identical names	Namesake

Our run found 13 kinds of error in all. The catalog we keep from these audits now holds 34, and it grows every run.

How do you run the audit on your own site?

Seven steps. Each one is something a person can do by hand, or hand to an agent.

1. Get the real page list. Read robots.txt for the sitemap location instead of assuming /sitemap.xml. Ours returned 404 at the default path. Include llms-full.txt if you have one, because that is the version agents read in one go.
2. Build a ledger of every number. Prices, durations, customer counts, response times. Write each one down with its URL. Any number that appears with two values is a finding.
3. Read the <head>, not just the page. Fetch pages with curl. Tools that summarize a page for a model often drop meta tags, canonicals and JSON-LD, which is exactly where a stray noindex or a mismatched description hides.
4. Compare machine files to the pages they point at. Every llms.txt line should match where its link lands today. A redirect that “works” but lands on a different page is still wrong.
5. Ask assistants about yourself twice. Once closed-book, with no hints, and once with search on. Give the closed-book test none of your facts. On our own run we accidentally did, and that test was worthless.
6. Read it as three hostile readers. A skeptical buyer, a competitor writing a battlecard, and someone trying to get an assistant to repeat a false claim about you. They find things the friendly read misses.
7. Verify every serious finding against the live page before you fix it. Re-fetch the page and find the quoted text. If it is not there, the finding is not real.

Step 7 is the one you cannot skip. More on why below.

How do you fix what you find without rewriting history?

Date the old page; do not delete it. On October 14 one pull request (22 files, 59 lines added, 127 removed) closed our worst problems:

• A visible “out of date” note. We added an optional superseded field to posts. It renders a note above the body, and in llms-full.txt, saying what changed. Five posts quoting retired prices now carry it. The original text stays, so the record is honest and the assistant reading it is warned.
• Labels a reader can see. The mock-up now says “illustrative” in visible text, not only in an attribute a screen reader announces.
• Fix the machine files. We removed a stale page, redirected the sitemap URL, and corrected the llms.txt lines.

The missing pricing page went live on October 13. The contradictions between pages are slower, because each one is a decision: which of two sentences is true now. That is the work the audit hands you. It cannot make it for you.

This is not the first time old prices bit us. Three weeks earlier we found four August guides pricing a ladder we had retired. If you change prices, sweep the archive the same day.

Can you hand the audit to AI agents?

Yes, as long as a person or a stricter model checks their work. We ran 15 read-only agents in 3 waves of 5: twelve each took one beat (pricing, founders, structured data, freshness, third-party profiles and so on), and three played the hostile readers. Each wave took about 2 to 4 minutes.

Four of the 15 made claims that failed verification. One said a disambiguation note appeared on none of our pages; it is on /about/. One called a retired package live; it had been marked deprecated about two weeks earlier. The adversary agent made three claims about things missing from the site, and all three were wrong. Five agents also wrote scratch files after being told not to. So the rule in our version is plain: nothing reaches the fix list until the coordinator has re-fetched the page and matched the quote.

Two more rules are now part of the audit. A 401, 403 or 429 from a link is “unknown”, not “dead”, because bot walls look like broken pages. And a deterministic link sweep now runs before any agent starts, so the agents spend their requests on judgment instead of checking links. How agents read a site behind those walls is its own story, in what happens when an AI agent hits a verify-you-are-human box.

What we’re still figuring out

• How long a fix takes to show up in what assistants say. We have not re-run the search-grounded test since the fixes.
• How to win back a name that other companies share. That part is mostly off your site.

For agents: try this yourself

• Find the contradictions. Read every page of [site], including old posts and llms.txt, and list each pair of sentences that would give an assistant two different answers to the same question, with both quotes and both URLs.
• Date the old claims. List every price, rate or offer on [site] with its URL and publish date, flag each one that disagrees with [current pricing page], and draft a one-sentence “out of date” note for each flagged page.

---

How this was made

Drafted by the Chronicler from the build sessions, commits and audit fix list behind this work, then edited and published by Chase.

See how the Chronicler works →

Read every page of [site], including old blog posts and llms.txt. List every pair of sentences that would give an AI assistant two different answers to the same question (price, who you serve, what you promise, how long something takes). For each pair give both quotes and both URLs. Do not suggest fixes yet.

List every price, rate, plan name or offer that appears anywhere on [site], with the URL and the page's publish date. Flag each one that disagrees with [current pricing page URL]. For flagged posts, draft a one-sentence "out of date" note that says what changed without deleting the original text.