Should I block AI crawlers?

It depends which one and which section, and it is a commercial decision rather than a technical one. Blocking a training crawler is defensible. Blocking a live retrieval crawler removes you from answers being generated today. Those are different choices.

Does Google-Extended stop me appearing in AI Overviews?

No. It controls whether your content may be used to train certain Google models. AI Overviews run on the ordinary Google index through Googlebot, so the only way out of them is being absent from Google search entirely.

What is the difference between GPTBot and ChatGPT-User?

GPTBot collects data that may be used for training. ChatGPT-User fetches a page live because someone asked a question right now, and OAI-SearchBot indexes for ChatGPT search. Blocking all three because they share a vendor is the common expensive mistake.

Can AI crawlers read JavaScript?

Less reliably than Googlebot, which has had years of investment in rendering. Treat anything absent from the initial HTML response as absent altogether, including content behind tabs, accordions and load-more buttons that fetch on click.

Does robots.txt remove content AI has already collected?

No. It is a request rather than enforcement, it does not bind operators who ignore it, and content already inside a trained model stays there. If you need legal control rather than a request, terms of service or authentication are the right instruments.

What is the single most useful technical change I can make?

Serve the substance in the initial HTML and confirm it with one curl command per page. If the phrase you want quoted is not in that output, no amount of content work matters, because the content is not present as far as retrieval is concerned.

05Module

Understanding AI Crawlers

How GPTBot, ClaudeBot, PerplexityBot, and other AI crawlers discover and index your content. What they look for and how to ensure access.

Available3 lessons19 min

AI crawlers do three different jobs, and confusing them is how sites make themselves invisible. Training crawlers collect data for future models, live retrieval crawlers fetch your page because someone asked a question right now, and search crawlers index for an AI product. Blocking the first is a defensible business decision. Blocking the second removes you from answers being generated today.

This module is the practical floor. Nothing else in the course matters if a crawler cannot reach your page or cannot read it once it arrives, and both failures are more common and more mundane than the strategy conversation suggests.

It separates the three jobs AI crawlers do, which is the distinction that decides your policy: blocking a training crawler is a defensible business decision, and blocking a live retrieval crawler removes you from answers being generated today. Most robots.txt files that block AI agents were written without that distinction in mind.

It then treats access as the commercial decision it is, and closes on the technical work that makes a page usable once it is fetched: substance in the initial HTML, honest heading structure, and provenance attached to anything you want repeated.

If you read one lesson: Optimizing for AI Discovery, and run the curl check in its first section before anything else on this site.

Lessons

Questions this module answers

Should I block AI crawlers?: It depends which one and which section, and it is a commercial decision rather than a technical one. Blocking a training crawler is defensible. Blocking a live retrieval crawler removes you from answers being generated today. Those are different choices.; In fullManaging AI Crawler Access
Does Google-Extended stop me appearing in AI Overviews?: No. It controls whether your content may be used to train certain Google models. AI Overviews run on the ordinary Google index through Googlebot, so the only way out of them is being absent from Google search entirely.; In fullMeet the AI Crawlers
What is the difference between GPTBot and ChatGPT-User?: GPTBot collects data that may be used for training. ChatGPT-User fetches a page live because someone asked a question right now, and OAI-SearchBot indexes for ChatGPT search. Blocking all three because they share a vendor is the common expensive mistake.; In fullMeet the AI Crawlers
Can AI crawlers read JavaScript?: Less reliably than Googlebot, which has had years of investment in rendering. Treat anything absent from the initial HTML response as absent altogether, including content behind tabs, accordions and load-more buttons that fetch on click.; In fullOptimizing for AI Discovery
Does robots.txt remove content AI has already collected?: No. It is a request rather than enforcement, it does not bind operators who ignore it, and content already inside a trained model stays there. If you need legal control rather than a request, terms of service or authentication are the right instruments.; In fullManaging AI Crawler Access
What is the single most useful technical change I can make?: Serve the substance in the initial HTML and confirm it with one curl command per page. If the phrase you want quoted is not in that output, no amount of content work matters, because the content is not present as far as retrieval is concerned.; In fullOptimizing for AI Discovery

Lessons

Questions this module answers

Should I block AI crawlers?: It depends which one and which section, and it is a commercial decision rather than a technical one. Blocking a training crawler is defensible. Blocking a live retrieval crawler removes you from answers being generated today. Those are different choices.; In fullManaging AI Crawler Access
Does Google-Extended stop me appearing in AI Overviews?: No. It controls whether your content may be used to train certain Google models. AI Overviews run on the ordinary Google index through Googlebot, so the only way out of them is being absent from Google search entirely.; In fullMeet the AI Crawlers
What is the difference between GPTBot and ChatGPT-User?: GPTBot collects data that may be used for training. ChatGPT-User fetches a page live because someone asked a question right now, and OAI-SearchBot indexes for ChatGPT search. Blocking all three because they share a vendor is the common expensive mistake.; In fullMeet the AI Crawlers
Can AI crawlers read JavaScript?: Less reliably than Googlebot, which has had years of investment in rendering. Treat anything absent from the initial HTML response as absent altogether, including content behind tabs, accordions and load-more buttons that fetch on click.; In fullOptimizing for AI Discovery
Does robots.txt remove content AI has already collected?: No. It is a request rather than enforcement, it does not bind operators who ignore it, and content already inside a trained model stays there. If you need legal control rather than a request, terms of service or authentication are the right instruments.; In fullManaging AI Crawler Access
What is the single most useful technical change I can make?: Serve the substance in the initial HTML and confirm it with one curl command per page. If the phrase you want quoted is not in that output, no amount of content work matters, because the content is not present as far as retrieval is concerned.; In fullOptimizing for AI Discovery