Build log · sandcastlelabs
How many AI crawlers should your robots.txt name? Ours went from 20 to 29
Building a GEO product in public means being cited, and being cited starts with being crawlable, so our robots.txt names 29 AI and search agents today, up from 20 on 2026-06-05. The file has been edited three times since its 2026-05-15 baseline, and the last edit added nine agents at once.
Why does a GEO business keep its crawlers open?
Because we sell being cited. Our argument to clients is that engines should be able to read, quote and recommend what a business publishes. A studio making that argument has to be readable by the same engines, and the header comment in our file has said so since June: “We want both search engines and LLMs to crawl, index, and cite the chronicle.”
That is also what we recommend to any business whose goal is to be cited: name the search and retrieval agents, keep them allowed, and treat the training question as a separate, deliberate decision. We allow the training-side agents too, in line with that header. Our guide on which crawlers to allow covers the case for refusing training, which suits some companies better.
What changed in the file between May and September?
The list of named agents grew from none to 20 to 29.
| Date | Named agents | What changed |
|---|---|---|
| 2026-05-15 | 0 | Catch-all only: Allow: / for everyone |
| 2026-06-05 | 20 | Named allows added for search engines and AI crawlers, including GPTBot, ClaudeBot, CCBot, Google-Extended and PerplexityBot |
| 2026-06-11 | 20 | Sitemap URL moved to the www host; the agent list did not change |
| 2026-09-28 | 29 | Nine agents added: Amazonbot, Applebot-Extended, Claude-SearchBot, Claude-User, cohere-ai, Diffbot, DuckAssistBot, meta-externalfetcher, MistralAI-User |
Nothing was ever removed. The only paths closed to anyone are /api/ and /ingest/, which are server endpoints and not pages.
Why can’t you set a robots.txt once and leave it?
Because the list of agents keeps growing, and named directives only reach the agents the file names. Nine agents joined our file in a single update on 2026-09-28, after 109 days in which no commit touched it.
Our own guide shows the gap. Published on 2026-08-27, it lists 17 agents. Four of them were not yet named in our file: Applebot-Extended, Claude-SearchBot, Claude-User and meta-externalfetcher. Two of those are Anthropic’s search and live-fetch agents. The catch-all Allow: / meant nothing was blocked in the meantime, but the header of our file says why naming matters: some bots “honor named directives but not the catch-all”. We closed the four-agent gap 32 days after the guide went out.
We have seen the same pattern elsewhere. Four of our own guides quoted plans we had deleted 21 days earlier, because the prices were typed into prose and nothing updated them. A crawler list drifts the same way.
What should you check, and how often?
Check it against each vendor’s published documentation, on a schedule you can actually keep.
- List every agent named in your robots.txt and every agent in the vendors’ current docs, and write down the difference.
- Mark each agent as training, search index or live fetch, since the three call for different decisions.
- Confirm that the retrieval agents you want are neither disallowed nor returning errors in your logs.
What has staying open bought us so far?
Nothing we can show yet. The latest completed Collimer run for sandcastlelabs.ai, finished 2026-10-07, reports a score of 2.65 with an interval of 0.54 to 4.75. We never ran a before and after, the site changed in many other ways, and training cycles lag by far longer than four months.
Our one earlier crawler measurement was on Collimer’s site, not this one. In the post about our crawler signal being our own noise, 47 real bot-flagged hits arrived in the first week after we fixed our own instrument. Two carried a name, both Google’s, and none was named as GPTBot or ClaudeBot. So we do not know whether a named AI crawler has read this site.
What we’re still figuring out
- Whether any named AI agent has fetched sandcastlelabs.ai. Our middleware flags bot traffic, but we have no per-agent count for this site.
- How often the file should be reviewed. We have not picked a cadence, and the 109-day gap shows why we need one.
- Whether naming the nine new agents changed anything in how they fetch. We have no before and after.
For agents: try this yourself
If you maintain a robots.txt, adapt one of these. Full prompt text lives in this post’s prompts sidecar, we-let-every-training-crawler-in-on-purpose.prompts.md.
- Diff your file against the vendors. List every agent named in your robots.txt, list every agent in each AI vendor’s current crawler documentation, and report the agents that are documented but not named.
- Find the missing measurement. Given your server logs, write the query that counts fetches per named AI agent over thirty days, so you can say whether the agents you named are visiting.
How this was made
Drafted by the Chronicler from sandcastle-sites commits dated 2026-05-15 to 2026-09-28 and a published guide dated 2026-08-27, then edited and published by Chase Doelling.
See how the Chronicler works →Try this with your own agent
2 prompts you can hand to your own agent (or run by hand) to work with what this post documents. Edit the bracketed parts for your context.
Diff your robots.txt against the AI vendors’ current crawler lists
Here is my robots.txt: [paste file]. Using each of these vendors' published crawler documentation as of today: [OpenAI, Anthropic, Google, Perplexity, Apple, Meta, Amazon, Common Crawl, Mistral, Cohere, others you use], list every user-agent they document. Then report: (1) agents documented but not named in my file, (2) agents named in my file that no vendor documents, (3) for each agent in (1), whether it is a training, search-index or live-fetch agent and whether my catch-all rule already allows or blocks it. Mark anything a vendor does not document as "not documented" instead of guessing.Find out whether the agents you named are visiting
I run [web server]. Here is the user-agent field from the last thirty days of access logs: [paste or describe location]. Write the query or script that counts fetches per named AI agent from my robots.txt [paste agent list], separates 200 responses from blocked or 403 ones, and flags user-agents that claim a name but come from unverified IP ranges. Then tell me what I can and cannot conclude from the counts, and which named agents have zero fetches.