See which AI crawlers a site's robots.txt allows or blocks, and whether it has an llms.txt.
It loads the site's live robots.txt once and, for each AI crawler below, finds the group that names it (or the * group when none does) and reports whether the home page is allowed, blocked, or partially blocked, with the rule that decided it. It also checks whether /llms.txt exists.
/ and makes no exceptions./ is allowed but /blog/ is not. Path rules under * that every crawler gets are noted, not counted.A crawler follows only the group that names it. If you write a GPTBot group, the rules under * no longer apply to GPTBot.
We ran this same check on the 10,000 most popular sites in October 2026. 84% block no AI training crawler, and AI search crawlers are blocked about half as often as training crawlers. Read the study.
Test any path against any crawler with the robots.txt tester, draft a file with the llms.txt generator, or see all free tools.
Fifteen user-agent tokens: GPTBot, OAI-SearchBot and ChatGPT-User (OpenAI); ClaudeBot, Claude-SearchBot and Claude-User (Anthropic); PerplexityBot and Perplexity-User; Google-Extended; Applebot-Extended; CCBot (Common Crawl); Bytespider; Meta-ExternalAgent; Amazonbot; and cohere-ai. Crawlers not on that list are not checked; use the robots.txt tester for any other user-agent.
No. robots.txt is a request, not enforcement. Well-behaved crawlers follow it, but nothing technically stops a client that ignores it. Operators of user-triggered fetchers, such as ChatGPT-User or Perplexity-User, may treat robots.txt differently because a person asked for the page; check each operator's documentation. This tool reports what the file says, not what any crawler actually does.
No. Google-Extended is not a crawler. It is a token that controls whether pages Googlebot crawls may be used for Gemini models. Google Search crawling is controlled by Googlebot, which this tool does not check; use the robots.txt tester for that.
A group that names the crawler allows the home page but disallows some paths, or a group blocks the home page with Allow exceptions. Path rules in the * group, which every crawler gets, are mentioned but not counted as a partial block. Test a specific path with the robots.txt tester.
It looks for a group whose User-agent line is exactly the crawler's token, ignoring case, and falls back to the * group when none matches. A group for Claude does not apply to ClaudeBot. Several groups naming the same token are combined, as Google documents for its own crawlers.
It requests /llms.txt on the same host and reports whether a text file is there. It does not check the file's format or content.