In short: AI crawlers are not one thing, and confusing them is how sites accidentally make themselves invisible. They fall into three groups with different purposes: bots that collect training data, bots that fetch pages live to answer a question someone is asking right now, and bots that index for a search product.
Blocking the first group is a defensible business decision. Blocking the second removes you from answers today. Most robots.txt files that block AI crawlers were written without that distinction in mind.
Three Jobs, Not One
Before any list of user agents, the distinction that actually decides your policy.
| Purpose | What it does | What blocking it costs you |
|---|---|---|
| Training collection | Gathers text to train future models | Long-term absence from what models know without searching. Nothing visible this year. |
| Live retrieval | Fetches your page right now because a user asked something | Immediate. You cannot be cited in answers you are not fetched for. |
| Search indexing | Builds an index that AI products query | Removal from that product's answers, and often from its conventional search too. |
The mistake this table exists to prevent: a team decides it does not want its content training future models, blocks everything with "AI" or "GPT" in the name, and removes itself from live citations at the same time. Those are different decisions with different costs, and one robots.txt line frequently makes both.
Who Is Actually Crawling You
OpenAI
OpenAI operates separate agents for separate purposes, which is what makes a precise policy possible.
- GPTBot collects data that may be used for training. This is the one to block if training is your concern.
- OAI-SearchBot indexes for ChatGPT's search feature. Blocking it removes you from that surface.
- ChatGPT-User fetches a page live when a user's question requires it. Blocking it removes you from answers being generated right now.
Blocking GPTBot while allowing the other two is a coherent position: keep my content out of training, keep me visible in answers. Blocking all three because they share a vendor is not.
Google separates its AI training access from Googlebot with the Google-Extended token. It is not a crawler in its own right: it is a control that tells Google whether your content may be used for training certain AI models.
The important detail, and the one most often misunderstood: Google-Extended does not control AI Overviews. Those draw on Google's ordinary index through Googlebot. If you want to be absent from AI Overviews, the only lever is being absent from Google search, which is a very different decision.
Anthropic
ClaudeBot is the primary crawler, and Anthropic has also used anthropic-ai and Claude-Web across different functions. Anthropic publishes its crawler documentation and honours robots.txt.
Perplexity
PerplexityBot indexes for search. Because Perplexity cites more heavily than any other major platform, blocking it forfeits more citation opportunity per unit of traffic than blocking anyone else.
Microsoft
Bingbot is the one that matters, and it is not an AI crawler at all. Copilot is grounded in Bing's index, so your Copilot visibility is entirely a function of your Bing indexing. Teams that have never looked at Bing Webmaster Tools are frequently invisible there for reasons unrelated to AI.
Common Crawl
CCBot is worth knowing about because it is upstream of everything else. Common Crawl is a non-profit that publishes an open web archive, and that archive has been a component of many training datasets. Blocking CCBot affects an unknown number of downstream users, present and future, which makes it a broader decision than blocking any single company's bot.
User agent lists go stale. The names above are correct as of this writing and will change: companies add agents, rename them and split them. Verify against the operator's own documentation before writing a rule, and check your server logs for what is actually hitting you. Do not copy a robots.txt from a blog post, including this one.
What They Can And Cannot Read
A practical limitation that changes what you build. AI crawlers are generally less capable at executing javascript than Googlebot, which has had years of investment in rendering.
Content that only exists after client-side rendering may simply be absent from what these crawlers see. So may content behind interaction: tabs, accordions and "load more" buttons that fetch on click. The safe assumption is that if it is not in the initial HTML response, it may not exist as far as an AI crawler is concerned.
You can check this in one command per page, with curl or any tool that fetches raw HTML without executing scripts. If your key answer is not in that output, it is not reliably available to retrieval.
Recognising Them In Your Logs
Your server logs are the only source of truth about who is visiting. Published lists tell you who exists; logs tell you who cares about you.
- Filter your access logs for the user agent strings above over the last 30 days.
- Count requests per agent. The distribution is usually uneven and often surprising.
- Look at which URLs each agent fetches. Heavy crawling of pages you consider secondary is a signal about what these systems think you are for.
- Check for agents you do not recognise. Not everything claiming to be an AI crawler is one, and user agent strings are trivially forged. Reverse DNS verification is the check that matters where an operator documents it.
- Compare crawl frequency against your publishing frequency. If a bot visits monthly and you publish weekly, most of your work is invisible to it between visits.
This is a twenty minute exercise, and it converts the policy question in the next lesson from an abstract debate into a decision about specific traffic you can see.
Key Takeaways
- AI crawlers do three different jobs. Training collection, live retrieval and search indexing carry very different costs when blocked.
- OpenAI separates GPTBot, OAI-SearchBot and ChatGPT-User, which makes a precise policy possible rather than all-or-nothing.
- Google-Extended controls training use only. It does not remove you from AI Overviews, which run on the ordinary Google index.
- Copilot visibility is Bing indexing. Check Bing Webmaster Tools before concluding anything about it.
- AI crawlers handle javascript less reliably than Googlebot. If your answer is not in the raw HTML, treat it as absent.
- Verify agent names against operator documentation and your own logs. Published lists, including this one, go stale.
Check yourself
Before you move on
Not scored, not recorded, and not part of the certificate. Both answers are settled by a sentence in this lesson, and the reasoning appears whichever option you pick.
- 01
Blocking which group of crawlers removes you from answers today?
- 02
What is the practical value of reading your own server logs first?