Your page can load perfectly in your browser and still reach ChatGPT as a 403, a "verify you are human" page or a shell with no text in it. robots.txt does not warn you: it can say every AI agent is welcome while a firewall rule two layers further down turns them away. You only find out by requesting the page the way an agent does and reading what comes back.
Check one page now with the free AI agent view checker. It requests the page once as a browser and once each as ChatGPT-User, GPTBot, PerplexityBot and ClaudeBot, and shows what each one received.
Why this matters now
One vendor case study shows what can happen when AI agents get a better page. Scrunch published it of Akamai's own site: from 1 December 2025 to 19 January 2026, Akamai served AI agents a server-rendered copy of some pages, with no JavaScript dependency and less page chrome, and compared those pages with pages that were not served that way. Scrunch reports:
| Metric (pages served to AI agents vs. pages not) | Change reported by Scrunch |
|---|---|
| Total AI citations | +85% |
| Unique prompts with citations | +38% |
| Brand presence on non-branded prompts | +364% |
| ChatGPT brand presence vs. competitors | +133% |
Read those numbers with their caveats. It is a vendor's study of its own product, about seven weeks long, the sample sizes are not published and the comparison pages were not chosen at random. Other outlets have quoted different figures for the same work. We would not promise anyone a number from it.
The direction is still worth taking seriously, because the cause is ordinary: AI agents often get a worse page than people do. Akamai solved it with a CDN product that most sites cannot buy. Most sites do not need to. The common causes are cheap to find and cheap to fix.
Three ways a page fails an AI agent
1. A firewall or CDN rule blocks or challenges the agent
Bot protection is built to stop automated traffic, and an AI agent is automated traffic. A CDN bot setting, a web application firewall rule, a rate limit or a WordPress security plugin can answer an agent's request with a 403 Forbidden, a 429 Too Many Requests, or a challenge page that asks the visitor to prove it is human. An agent cannot solve a challenge, so it never reaches your content. A single bot setting can cover AI agents along with scrapers, and it is easy to forget that it does.
You will not see this in a browser. Your browser passes the checks, so the page looks fine.
2. robots.txt asks the agent to stay out
The polite version of a block. Each AI company publishes the user-agent tokens its crawlers and fetchers use, and robots.txt rules are addressed to those tokens: OpenAI's crawler overview lists GPTBot (training) and ChatGPT-User (fetches a page when a user's request needs it), and Perplexity's lists PerplexityBot. A Disallow: / under one of those tokens asks that agent to stay out, whether or not anyone meant it to.
Most sites do not do this on purpose. When we read robots.txt on the Tranco top 10,000 sites on 17 October 2026, 84.2% of the 6,894 sites that gave a readable answer blocked no AI training crawler at all. That is the point of this post: robots.txt saying "allowed" is the start of the question, not the answer.
3. The content only exists after JavaScript runs
Many sites send a nearly empty HTML document and build the page in the browser with JavaScript. A person never notices. AI agents generally read the HTML the server sends and do not run the page's scripts, so they receive the empty document: a title, a few menu words and a <div id="root">.
This is the cause the Akamai setup addressed: the copy it served to agents was server-rendered, with no JavaScript dependency.
How to check what AI agents receive
Free, one page at a time: the AI agent view checker fetches the page as a browser and as each of the four agents, using each agent's published user-agent string, and shows per agent the HTTP status, whether it got a block or a challenge page, what your robots.txt says about it, how many words it received and what share of the browser's text that is. It finishes with a verdict and a fix list. It does not render the page, so it cannot see content that appears only after JavaScript; when the HTML has almost no text, it says the page is probably built by JavaScript.
By hand: request the page with an agent's user agent and look at the status and the body.
curl -s -o /dev/null -w "%{http_code}\n" \
-A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot" \
https://www.example.com/
A 200 is a start. Then fetch the body and check that your main text is in it, not a challenge page and not an empty shell.
With the API: GET /agent-view (Starter plan and up) does all of the above and also renders the page in a headless browser, so it reports content that only appears after JavaScript runs. It is available as the agent_view tool on the MCP server, and ?include=agent_view adds it to any paid /audit call.
One caveat applies to every method, ours included. A request from our server or your laptop that sends ChatGPT-User's user agent is not ChatGPT. A site that verifies agents by network address may answer the real one differently, in either direction. Treat a block found this way as a strong signal, then confirm it in your CDN or server logs by filtering for the agent's user agent and looking at the status codes you returned.
How to fix it
If an agent is blocked or challenged: find the rule and make an exception for the agents you want to reach your content. Look in your CDN or firewall's bot settings first (in Cloudflare that is under Security, Bots, including the AI bots setting and any custom WAF rule), then rate limits, then your server configuration and any security plugin. Most bot-protection products have a verified-bots or AI-bots option, which is better than allow-listing a user-agent string anyone can copy.
If robots.txt disallows an agent: decide whether you mean it. Keeping training crawlers out while letting ChatGPT-User and PerplexityBot in is a reasonable choice, and many sites make it. If the rule is an accident, remove it or narrow it to the paths you want kept private. The free AI crawler checker shows what your robots.txt says to 15 AI crawlers.
If the content needs JavaScript: put the main content in the HTML the server sends. Server-side rendering, static generation or pre-rendering in your framework all do it; most modern frameworks support at least one. You do not need a second site for AI agents. Serving everyone the same complete HTML is simpler to maintain and avoids the risk of showing crawlers something different from what people see, which Google's spam policies on cloaking treat as a violation. (Akamai's setup, as described in the case study, kept people and search engine crawlers on the normal page.)
How to tell whether the fix worked
Run the agent view check again after the change. Then measure what the change was for: whether AI assistants mention and cite you more often.
That is what an AI citation tracker measures, and it now takes a change date. Mark the day the fix went live and it compares the runs before with the runs after, per engine, with the counts behind every rate (GET /citations/trackers/{id}/history?change_date=2026-11-03&runs=4, or the "Before and after a change" section on the AI Visibility dashboard). It tells you plainly when there are too few runs on either side to compare, and whether the new rate falls outside the range the earlier runs covered on their own. AI answers vary from run to run, so a few points of movement after one run proves nothing; several runs on each side do.
What this check does not tell you
- Whether an AI assistant will cite you. Reaching the page is necessary, not sufficient. Content, authority and what other sites say about you still decide that.
- What the real agent received. Our requests come from our server; see the caveat above.
- Anything about pages you did not check. Bot rules are often path-specific. Check your key pages, not just the homepage.
FAQ
Can ChatGPT read my website?
It can read what your server sends to its user agent, if your robots.txt and firewall let it through. Check with the free AI agent view checker, then confirm a block in your server logs, because our requests are not ChatGPT's own.
Why does my site block AI bots when I never set that up?
Bot protection on CDNs, firewalls and security plugins treats AI agents as automated traffic, and some products group them under one setting. A rate limit can also catch an agent that requests several pages quickly.
Should I serve AI agents a different version of my page?
We do not recommend it. Serve everyone the same complete HTML. A separate copy for agents is more to maintain and, done badly, drifts toward cloaking.
Does robots.txt "allowed" mean AI agents can read my page?
No. robots.txt is a request about what an agent should fetch. It says nothing about whether your firewall lets the agent in, or whether the HTML it receives has your content in it.