We Read robots.txt on the Top 10,000 Sites. 84% Block No AI Training Crawler.

On 17 October 2026 we fetched robots.txt from the 10,000 most popular sites on the web and asked one question per site, fifteen times over: is this AI crawler allowed to fetch the homepage?

The short answer is that almost everyone still lets them in. Of the sites that gave us a readable answer, 84.2% block no AI training crawler at all, and only 4.2% block all nine training crawlers we checked.

Run the same check on your own site with the free AI crawler checker. It uses the exact rules this study used, so your result and these numbers can be compared directly.

What we measured

• Sites: the Tranco top 10,000 list, downloaded the day of the run.
• Crawlers: 15 user-agent tokens in three groups. Nine collect training data (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, Meta-ExternalAgent, Amazonbot, cohere-ai). Three build AI search indexes (OAI-SearchBot, Claude-SearchBot, PerplexityBot). Three fetch a page when a user asks an assistant to (ChatGPT-User, Claude-User, Perplexity-User).
• Path: /, the homepage. A site that blocks only /private/ counts as allowing the crawler.
• Rules: a crawler follows the group that names its token, or * when none does. "Blocked" means that group disallows /. "Partial" means the group names the crawler and mixes allow and disallow rules.

6,894 sites gave a readable answer: either a robots.txt file or no file at all (a 404, which means everything is allowed). The other 3,106 were left out of every percentage. Some refused us with a 401, 403 or 429 or returned a server error, and many Tranco entries are CDN, API and tracking domains that have no website to read. Every percentage below is out of the 6,894.

Finding 1: training crawlers are blocked twice as often as AI search crawlers

The gap is the story. Sites are much more willing to say no to a bot that takes content for model training than to one that might send them a visitor from an AI answer.

6.0% of sites block at least one training crawler while allowing every AI search and user-triggered crawler. That is the "don't train on me, but do cite me" position, written out in robots.txt.

Finding 2: the most blocked crawlers are the old, well-known ones

Share of sites blocking each crawler from the homepage:

Bytespider, CCBot and GPTBot have been named in blocklists for years, and it shows. The crawlers at the bottom are newer or are the ones that bring traffic back.

Finding 3: blocklists go stale

2.1% of sites block GPTBot but let ClaudeBot through. Most of those are not making a choice about Anthropic. They copied a list of AI user agents at some point, and the list did not have every crawler on it, or a crawler appeared after they wrote it.

A blocklist is only as current as the last time someone edited it. If your goal is to keep training crawlers out, a list that names five of them leaves the rest allowed by default. If your goal is to stay visible in AI answers, check that a broad rule hasn't caught the search and user-triggered crawlers by accident.

Finding 4: the biggest sites block more

The top 1,000 is where the publishers, retailers and platforms with licensing deals and legal teams sit, and they are more likely to have taken a position.

Finding 5: robots.txt is nearly universal, llms.txt is not

83.9% of sites served a robots.txt file. The rest had none, which allows every crawler.

14.1% answered /llms.txt with something other than an HTML page. Treat that as an upper bound: we checked that the path returned a non-HTML file, not that the file follows the llms.txt format. Some servers answer every unknown path with plain text. If you want one, our llms.txt generator builds a draft from your home page and sitemap.

What this study does not tell you

• robots.txt is a request, not a wall. We measured what sites ask for, not what any crawler does. Some sites block AI crawlers at the firewall instead, and those blocks don't show up in robots.txt.
• One path. We tested /. A site that allows the homepage but blocks its archive or product pages is counted as allowed.
• Fifteen tokens. New crawlers appear often. A site that blocks one we didn't list is not counted for it.
• One day. These files change. We will rerun the study and publish the comparison.

How do I check which AI crawlers my site blocks?

Paste your URL into the AI crawler checker. It fetches your robots.txt, shows a verdict for each of the 15 crawlers in this study and the rule that decided it, and checks for an llms.txt file. It's free and needs no signup.

Should I block AI training crawlers?

It depends on what your content is worth to you as training data compared with the reach AI products can give you. Most sites in this study haven't blocked them. If you do block them, name each training crawler separately and leave the search and user-triggered ones allowed, so you can still appear and be cited in AI answers.

Does blocking GPTBot stop my site appearing in ChatGPT answers?

Not by itself. GPTBot collects training data. ChatGPT's search results come from OAI-SearchBot, and pages it fetches while answering a user come from ChatGPT-User. Each one follows its own rules in robots.txt, so you can block one and allow the others.

Method and code

The survey ran on a GitHub Actions runner, 64 connections at a time, with the user agent SEOScoreAPI-Research/1.0. It made one request for robots.txt (trying the www. host if the bare domain didn't answer) and one for /llms.txt, and did nothing else. The verdict for each crawler comes from the same function as the free checker, so the tool and these numbers can't drift apart. The run took 15 minutes.

If you want to know whether AI answer engines actually mention your brand once they can read your site, that's what AI citation tracking measures.