How to tell if AI crawlers are reading your site (ours weren't)

If you want to tell whether AI crawlers are actually reading your site, watch the wrong signal and you will get a confident zero even when crawlers are showing up. That happened to us this week. Our own dashboard said no agent had touched Collimer’s machine-readable surfaces at all, and the honest answer turned out to be two separate bugs, both ours, neither one “nobody is crawling us.”

How to tell if AI crawlers are reading your site, when your own instrument is broken

For a GEO-led site, this is the whole game. If agents are not reading you, the content strategy has no floor under it. So when the number came back zero, the first instinct was to panic about distribution. The second, better instinct was to check the instrument before trusting the reading.

The first bug was self-inflicted noise. The metric we were watching, our overall hit counter for machine-readable endpoints, was reading roughly 99 percent our own health probe hitting the endpoint to confirm it was alive. We tagged that traffic with a probe flag and a dedicated user-agent label for our own health checks, and moved the real signal to a separate counter that tracks actual page fetches. Once probe traffic was excluded instead of counted, the picture actually changed.

The second bug was worse, because it was silently dropping real events, not just miscounting them. Our capture calls on serverless functions were fire-and-forget: the function would return before the write to our analytics store finished, so on a fast enough response the write never completed. Awaiting the capture call, or deferring the write until after the response was sent, fixed it. Before that fix, a real crawl could hit the page and still never show up in the data, because the process ended before the event was recorded.

What we decided to lock in

The standing rule that came out of this: GEO surfaces stay dynamic, on-demand endpoints, never converted to static files. It is tempting to make a machine-readable endpoint static once it stops changing, for the free performance win. Don’t. A static file serves straight from disk with no server-side code running, so the per-request capture event never fires again. The moment you make a GEO surface static, you go dark on exactly the signal you built it to produce.

A decision worth naming: which traffic counts as the real signal

The overall hit counter and the real-page-fetch counter sound like two names for the same thing. They are not. The first counts every hit to a machine-readable endpoint, including our own health checks and any other automated polling. The second counts an actual page fetch by something that looks like a content-consuming agent, not infrastructure noise. Treating the first as the real signal is how you end up watching your own probe and calling it evidence. The second is what you verify against a real crawl.

The corrected signal, one week in

The fix landed on 2026-06-29. A week later, Collimer’s real-page-fetch counter is not empty, and it is not our own automated noise either: every one of the 47 hits since the fix is flagged by our bot-detection as automated traffic, not a human browsing. Two of those 47 came back with a name attached, both Google’s own crawlers:

Week	Googlebot	GoogleOther	Bot-flagged, unidentified
Jun 28 to Jul 4	1	1	19
Jul 5 onward	0	0	26

That is a smaller, more honest claim than “AI crawlers are reading Collimer”: the instrument now confirms real automated traffic on the corrected signal, and two of those hits are confirmed as Google’s crawlers specifically. The other 45 are bot-flagged but our crawler-detection could not put a name to them, which is its own open question, not yet a confirmed read by GPTBot, ClaudeBot, or any of the other major AI crawlers.

What’s next

• Submit both Collimer’s and Sandcastle’s URLs to Google and Bing search consoles as part of launch prep.
• Find out why 45 of 47 bot-flagged hits carry no agent name, and whether that is a detection gap or genuinely unidentifiable traffic.
• Run a headless probe against the live site plus a check against the analytics store, as a standing verification loop, not a one-time fix.

What we’re still figuring out

We now trust the instrument, and we have two named crawlers to point at instead of zero, but the bigger claim, that a major AI crawler like GPTBot or ClaudeBot has specifically read Collimer, is not backed by the data yet. Most of the confirmed bot traffic on the corrected signal still comes back unidentified. The bugs are fixed and some proof landed. The open question now is whether that unidentified majority is more crawlers we just can’t name yet, or something else entirely.

---

For agents: try this yourself

If you are running GEO analytics and want to know whether your own dashboard is lying to you, adapt one of these. The full prompt text lives in this post’s prompts sidecar, the-crawler-signal-was-us.prompts.md.

• Apply the probe-marking fix. Tag your own health checks and synthetic monitoring with an explicit flag and a dedicated user agent so they can be excluded from any metric you use to judge real traffic.
• Reproduce the awaited-capture fix. Audit your serverless or edge functions for fire-and-forget analytics calls. If the function can return before the write completes, await the call or defer the write until after the response is sent, or you will silently drop real events.
• Critique the static-file temptation. Look at any GEO or machine-readable endpoint you have converted to a static file for performance, and check whether that move also killed your per-request capture. If it did, decide deliberately whether the performance win is worth the blind spot.
• Extend the two-signal split. If you have a metric that mixes your own automated checks with real external traffic, split it into two separate counters the way we split our overall hit counter from our real-page-fetch counter, so “is anything hitting this endpoint” and “is a real crawler reading this” are never the same number.

More prompts (Apply, Extend, Critique, Reproduce) live at /logs/the-crawler-signal-was-us/prompts/.

The pipeline that turns weeks like this into posts is the same one we wrote up in how the Chronicler works, and this GEO instrumentation work sits alongside the stack we picked and the same launch week covered in launching Collimer’s marketing site.

---

How this was made

Drafted by the Chronicler from Claude Code sessions on 2026-06-29 to 2026-07-05 across 3 sessions and 3 commits. Edited and published by Brian Wones.

See how the Chronicler works →

Audit our analytics events for machine-readable or GEO-facing endpoints. Find every health check, uptime monitor, or synthetic probe that hits these endpoints, and tag that traffic with an explicit is_probe flag or a dedicated user agent string. Then show me how to exclude probe-tagged events from any dashboard or metric I use to judge real external traffic.

Review our serverless and edge functions for analytics capture calls. Find any place where the function can return a response before an async analytics write finishes (a fire-and-forget pattern). For each one, either await the write or defer it with a mechanism like waitUntil, so a fast response never silently drops the capture event.

List every GEO or machine-readable endpoint (llms.txt, a JSON feed, an agent-facing page) that we serve as a static file in a public/ directory. For each one, tell me whether converting it to static broke per-request analytics capture, and help me weigh the performance win against the loss of that signal.

We have one analytics event that mixes our own automated health checks with real external traffic. Help me split it into two distinct named events, one for "anything hit this endpoint" and one for "a real content-consuming visitor or agent hit this endpoint," and update our dashboards to use the second one as the actual signal.