In short: Two settings on a WordPress site can remove it from AI answers entirely, and both are quiet enough that teams run for years without noticing.
The Search Engine Visibility checkbox no longer writes anything into robots.txt. Since WordPress 5.3 it outputs a noindex meta tag into the page head instead, which is a different mechanism with a different reach.
And WordPress only serves its generated robots.txt when no real file exists. If somebody uploaded one, every plugin writing rules to the virtual file is being silently ignored.
The Checkbox That Changed What It Does
Settings, Reading, "Discourage search engines from indexing this site" is one tick box with a history worth knowing.
Until WordPress 5.2, ticking it made robots.txt return Disallow: /. From version 5.3 that stopped: it now emits <meta name="robots" content="noindex,nofollow"> into the head through wp_head, and leaves robots.txt alone.
Three consequences follow, and the third is the one that catches people.
The rule is now in the page rather than in a file, so it applies per page and only where the theme calls wp_head. A crawler has to fetch the page to learn it is unwelcome, which means the request still happens. And anybody auditing by reading robots.txt will find nothing wrong.
Check this before the logs and before anything else. It gets ticked during a rebuild, on a staging site that later becomes production, or by a developer who left, and none of those leave a trace anywhere else. It costs one click to check, and when it is the answer it is the whole answer.
The robots.txt That Is Not A File
WordPress generates robots.txt dynamically. There is no file on disk, and plugins add rules by filtering that generated output.
The behaviour worth knowing is what happens when a real file exists. A physical robots.txt in the site root is served by the web server before WordPress is ever involved, so the generated one never runs. Every rule your SEO plugin believes it is publishing goes nowhere, and the plugin's own settings screen will keep showing them.
The same applies when WordPress is installed in a subdirectory: the generated file is served from that subdirectory, and the one that matters is at the domain root, where WordPress is not.
How To Tell Which One You Have
Fetch it and compare it with what your plugin claims:
curl -s https://example.com/robots.txtA default generated file disallows the admin directory and allows admin-ajax.php. If what comes back does not resemble that and does not match your plugin's settings screen either, you are looking at a physical file somebody uploaded, and it is the one in charge.
Then read it against Meet the AI Crawlers. A file written years ago to block scrapers frequently names agents that now do live retrieval, which removes the site from answers being generated today.
Neither setting blocks anything. A robots rule is a request that the major operators honour, and a noindex tag is an instruction to the systems that read it. Both are conventions rather than enforcement, and neither removes content already collected.
The Third Thing Worth Checking
Not a setting, but it belongs with them: whether your host or security plugin is blocking AI agents outright.
Managed hosts and firewall plugins ship bot rules, and those lists are updated by the vendor rather than by you. An agent can be challenged or refused at the edge without appearing anywhere in your WordPress settings, and the only place it shows is the logs.
Filter thirty days of access logs for the agents that matter and look at the status codes. A run of 403s answers a question that no amount of configuration review would have.
Key Takeaways
- Since WordPress 5.3 the Search Engine Visibility checkbox emits a noindex meta tag and no longer touches robots.txt, so an audit that reads robots.txt alone will miss it.
- It is left over from a rebuild or a staging site often enough to be worth checking first, and it costs one click.
- WordPress generates robots.txt only when no real file exists. A physical file wins and silently ignores every plugin rule.
- Fetch robots.txt and compare it against what your plugin claims to be publishing. A mismatch tells you which one is live.
- A host or security plugin can refuse agents at the edge with nothing shown in WordPress. Only the logs reveal it.
Check yourself
Before you move on
Not scored, not recorded, and not part of the certificate. Both answers are settled by a sentence in this lesson, and the reasoning appears whichever option you pick.
- 01
Since WordPress 5.3, what does ticking "Discourage search engines from indexing this site" actually do?
- 02
An SEO plugin shows crawler rules in its settings, but the live robots.txt does not contain them. What is the most likely explanation?