robots.txt Compliance Suite: 40 checks | SEO Score API check catalog

ID	Check	Passes when	Weight	Runs
2.1.01	Fetch /robots.txt (raw HTTP)	HTTP 200	Critical	Deterministic
2.1.02	Ensure robots.txt is not returning 404	Exists	Critical	Deterministic
2.1.03	Validate robots.txt is under 500KB	Yes	Heavy	Deterministic
2.1.04	Validate robots.txt content-type is text/plain or text/plain; charset=utf-8	Correct MIME	Heavy	Deterministic
2.1.05	Check for syntax errors in robots.txt	No syntax errors	Heavy	Deterministic
2.1.06	Identify "Disallow: /" for User-agent: *	Not allowed unless intentional	Critical	Deterministic
2.1.07	Ensure homepage URL is allowed by robots.txt	Allowed	Critical	Deterministic
2.1.08	Ensure main service pages are allowed	Allowed	Critical	Deterministic
2.1.09	Ensure blog pages are allowed	Allowed	Critical	Deterministic
2.1.10	Identify any blocked CSS files	No blocks	Heavy	Deterministic
2.1.11	Identify any blocked JS files	No blocks	Heavy	Deterministic
2.1.12	Identify any blocked image directories	No blocks	Heavy	Deterministic
2.1.13	Identify any blocked /wp-content/ directories (WordPress)	Should not block core assets	Heavy	Deterministic
2.1.14	Identify any blocked /uploads/ directory	Not blocked	Heavy	Deterministic
2.1.15	Identify broken or disallowed JavaScript frameworks	Allowed	Heavy	Deterministic
2.1.16	Validate formatting of Allow: and Disallow:	Correct usage	Medium	Deterministic
2.1.17	Check if sitemap is referenced in robots.txt	Sitemap present	Critical	Deterministic
2.1.18	Validate sitemap URL in robots.txt returns 200	200	Medium	Deterministic
2.1.19	Validate sitemap URL uses HTTPS	HTTPS	Medium	Deterministic
2.1.20	Count number of Sitemap: entries	Reasonable (< 10)	Medium	Deterministic
2.1.21	Validate wildcard pattern support (*)	Valid usage	Medium	Deterministic
2.1.22	Validate end-of-line comments (#)	No interference	Medium	Deterministic
2.1.23	Validate blank lines interpretation	No malformed spacing	Medium	Deterministic
2.1.24	Validate no disallow for /wp-admin/admin-ajax.php (WordPress)	Not disallowed	Heavy	Deterministic
2.1.25	Validate no conflicting Allow/Disallow pairs	No conflicts	Heavy	Deterministic
2.1.26	Validate no-case sensitivity violations	Correct casing	Heavy	Deterministic
2.1.27	Validate robots.txt does not exceed 10,000 lines	Yes	Medium	Deterministic
2.1.28	Validate robots.txt does not block JSON endpoints	Allowed	Medium	Deterministic
2.1.29	Identify disallow of /api/ or /rest/ if site uses JS frameworks	Avoid blocking critical APIs	Medium	Deterministic
2.1.30	Validate no disallow rules match canonical URLs	True	Heavy	Deterministic
2.1.31	Identify excessive Allow: rules (>= 20)	Reasonable number	Medium	Deterministic
2.1.32	Identify excessive Disallow: rules (>= 20)	Reasonable number	Medium	Deterministic
2.1.33	Detect empty wildcard rules (Allow: *)	Avoid unnecessary wildcard	Medium	Deterministic
2.1.34	Validate that URL-encoded characters (%) are correct	No malformed encodings	Medium	Deterministic
2.1.35	Detect robots.txt cloaking (serving different robots.txt to Googlebot vs browser)	Identical	Critical	Deterministic
2.1.36	Validate no "Disallow: /*?" rules that break query URLs	Avoid unless intentional	Heavy	Deterministic
2.1.37	Detect if robots.txt is served with caching headers	Cache-Control present	Heavy	Deterministic
2.1.38	Validate robots.txt size stability (same file across repeated loads)	Stable	Medium	Deterministic
2.1.39	Ensure robots.txt loads consistently across user-agents	Identical for Chrome, Googlebot, Bingbot	Critical	Deterministic
2.1.40	Compute total robots.txt compliance composite score	>= 90%	Critical	Deterministic

2.1 robots.txt Compliance Suite

Run these checks on your site