Which AI crawlers visit a young site, and what do they read?

In the 30 days to October 7, five named AI crawlers made 209 fetches on our site. The eight most-fetched pieces were build logs and essays, and they drew 70 fetches between them. None of the eight linked to our AI visibility page or our work-with-us page. Here is what the counts showed and what you can check on your own site.

Which AI crawlers visited, and how often?

An AEO and GEO review of our own site on October 7 pulled the post_page_fetched event from PostHog, with our own probes excluded. The issue that came out of it lists five named crawlers:

Crawler	Fetches in 30 days
ClaudeBot	57
PerplexityBot	54
YouBot	48
GPTBot	27
Applebot	23

That is 209 fetches across those five. All five showed up, and the busiest drew about two and a half times the fetches of the quietest. These are small numbers. A site this young does not get crawled at volume, so each fetch is a visible choice about what a crawler thought was worth reading.

What did they read?

Six of the eight most-fetched pieces were build logs and two were essays. The most-fetched drew 15 fetches, the eighth drew 6, and the eight together drew 70.

Piece	Fetches
How an Agent-Native Studio Found Its Own Broken Links	15
The Price Is Public Because the Sprint Is Guided	10
We stopped selling Collimer as its own product (essay)	9
Last Week’s Bet Was Tested, and It Held	9
Week one: V1 of the site, and the parts that are not done	7
Build the machine, then hand someone the keys (essay)	7
How Does a One-Character Bug Affect Your AI Visibility Score on GEO Consulting Platforms?	7
Should You Hire a GEO Specialist, Use AI Consulting Tools, or Let an Agency Handle It?	6

What they share is that they are our record. They describe what we did, what we decided and why. None of them is a page that says what we sell, and neither of our commercial pages made the most-fetched list.

What can you learn from your own crawler counts?

Three checks, each cheap to run:

1. Count fetches by named crawler. Pick one event, one window and the crawlers you care about, and exclude your own test traffic. We did not know whether we would see one crawler or five until we looked.
2. List the pages that draw the fetches. For us the answer was the account of what happened, not the page about the offer. Your list may differ, and you will not know until you pull it.
3. Read what those pages link to. A crawler that reads a post sees whatever the post points at. We cannot see what a crawler does after it reads, but we could see that nothing on these eight pages pointed anywhere commercial.

What did we change after we saw it?

We annotated. Each of the eight got a short section headed “Where this went”, dated October 9, with one link in the body. The posts are otherwise unchanged. A published log is a record of what we believed on the day, and editing the argument to carry a link would change the record to serve the link.

• Three diagnostic pieces link to the AI visibility page.
• Two offer pieces link to the work-with-us page.
• Three the review had not mapped also link to work-with-us. Those were a judgment call and are the ones most open to a different choice.

The same pull request moved the “Method” line in our llms.txt from a methodology page on the old Collimer domain to our own methodology page. The site is one brand now, and a file written for machines should say so.

Did it work?

We don’t know. The pull request merged on October 9, and the follow-up check in the issue is a fortnight later: whether the AI visibility page and the work-with-us page show up in the crawler paths. That check falls on or after October 23. If they still do not appear, annotation was the wrong lever and we will say so here.

What we’re still figuring out

• Why crawlers favor the logs. They may be better linked, more recently updated, or simply easier to read. We have the counts and no cause.
• Whether one link in a dated note is enough weight for a crawler to follow. Position and wording may matter.
• Whether 70 fetches is signal at all. It is a small sample from a short window.
• Whether the five named crawlers are the whole picture, and whether the per-page counts include any traffic that is not an AI crawler. We are working from the figures in the issue, not a fresh query.

For agents: try this yourself

If you run a site that publishes its own work, adapt one of these. Full prompt text lives in this post’s prompts sidecar.

• See which AI crawlers visit your site and what they read. Ask your analytics agent for fetch counts by named AI crawler and the most-fetched paths over 30 days, then list what each of those pages links to.
• Annotate instead of rewriting. Draft a short, dated “Where this went” note for each page you want to point somewhere, with one link, and leave the original argument untouched.

---

How this was made

Drafted by the Chronicler from a GitHub issue, the pull request that closed it and the review that prompted both, then edited and published by Chase Doelling.

See how the Chronicler works →

Using [analytics tool], give me fetch counts by crawler, and the most-fetched paths on [our site], over the last 30 days for these named AI crawlers: [ClaudeBot, PerplexityBot, GPTBot, Applebot, YouBot]. Exclude our own test traffic. For each of the top [8] paths, check whether the page links in its body to [commercial page 1] or [commercial page 2], and report the count of such links. Tell me which of the two commercial pages, if either, appears in the top fetched paths. Do not infer why crawlers prefer a page; report only what the data shows.

For each of these published pages: [list]. Draft a short section headed "Where this went", dated [today], containing one sentence and one in-body link to [the matching commercial page]. Leave the original text unchanged. Use only claims about the commercial page that are true today, and flag any claim you could not source. Return each note separately so I can review them one at a time.