Home / Check catalog / 6. Schema, Local, Trust, Deep Audit / 6.35 Technical SEO Foundations / Crawlability / Indexability / Sitemaps / Robots / AI-Crawl Signals

6.35 Technical SEO Foundations / Crawlability / Indexability / Sitemaps / Robots / AI-Crawl Signals

200 checks in Schema, Local, Trust, Deep Audit. Automated Its subsection has a live automated checker in the engine.

robots.txt — Control, Safety, AI-Friendliness

ID	Check	Passes when	Weight	Runs
6.35.01	Validate robots.txt accessible at /robots.txt	Accessible	Critical	Deterministic
6.35.02	Validate robots.txt returns HTTP 200	200	Critical	Deterministic
6.35.03	Validate no global Disallow: / unless intentional	Safe	Critical	Deterministic
6.35.04	Validate correct User-agent blocks	Correct	Heavy	Deterministic
6.35.05	Validate robots.txt includes sitemaps	Included	Heavy	Deterministic
6.35.06	Validate no conflicting rules across user agents	Consistent	Heavy	Deterministic
6.35.07	Validate no deprecated syntax in robots.txt	Clean	Medium	Deterministic
6.35.08	Validate robots.txt not blocking assets needed for rendering	Allowed	Medium	Deterministic
6.35.09	Validate no accidental disallow of staging folders	Clean	Medium	Deterministic
6.35.10	Validate AI-friendly agents not blocked (ChatGPT, Claude, etc.)	Allowed	Medium	Deterministic

Sitemap Hygiene — XML, Image, Video, News, Hreflang

ID	Check	Passes when	Weight	Runs
6.35.11	Validate sitemap.xml accessible	Accessible	Critical	Deterministic
6.35.12	Validate sitemap.xml returns 200	200	Critical	Deterministic
6.35.13	Validate no 404 URLs inside sitemap	None	Critical	Deterministic
6.35.14	Validate only indexable URLs included	Correct	Heavy	Deterministic
6.35.15	Validate lastmod fields accurate	Accurate	Heavy	Deterministic
6.35.16	Validate image sitemaps correct (if present)	Correct	Heavy	Deterministic
6.35.17	Validate video sitemaps correct (if present)	Correct	Medium	Deterministic
6.35.18	Validate hreflang sitemaps correct	Correct	Medium	Deterministic
6.35.19	Validate sitemap index not missing children	Complete	Medium	Deterministic
6.35.20	Validate sitemap file size within limits	Within limits	Medium	Deterministic

Indexability — Meta Robots, Canonicals, Directives, Status Integrity

ID	Check	Passes when	Weight	Runs
6.35.21	Validate noindex/nofollow rules correct	Correct	Critical	Deterministic
6.35.22	Validate canonical tag present & correct	Correct	Critical	Deterministic
6.35.23	Validate canonical not self-contradicting	Consistent	Critical	Deterministic
6.35.24	Validate no duplicated canonical URLs across pages	Unique	Heavy	Deterministic
6.35.25	Validate hreflang tags correct	Correct	Heavy	Deterministic
6.35.26	Validate x-default exists for multilingual sites	Present	Heavy	Deterministic
6.35.27	Validate meta robots matches robots.txt behavior	Matched	Medium	Deterministic
6.35.28	Validate pages return indexable HTTP 200	Indexable	Medium	Deterministic
6.35.29	Validate pagination markup (rel=next/prev) correct	Correct	Medium	Deterministic
6.35.30	Validate canonical not pointing to redirects	Clean	Medium	Deterministic

HTTP Status Mapping — 200, 301, 302, 404, 410, 500

ID	Check	Passes when	Weight	Runs
6.35.31	Validate indexable URLs return 200	200	Critical	Deterministic
6.35.32	Validate 404 pages return actual 404 status	404	Critical	Deterministic
6.35.33	Validate 500 errors absent	None	Critical	Deterministic
6.35.34	Validate redirects use 301 for permanent	301	Heavy	Deterministic
6.35.35	Validate no redirect loops	None	Heavy	Deterministic
6.35.36	Validate no redirect chains > 2 hops	<2	Heavy	Deterministic
6.35.37	Validate 410 used for permanently removed content	Correct	Medium	Deterministic
6.35.38	Validate soft 404s not present	None	Medium	Deterministic
6.35.39	Validate 302 not used for permanent changes	Correct	Medium	Deterministic
6.35.40	Validate no 200 served for missing pages	Correct	Medium	Deterministic

Crawlability — Depth, Rendering, Loops, Faceted Navigation

ID	Check	Passes when	Weight	Runs
6.35.41	Validate crawl depth < 5 levels	<5	Heavy	Deterministic
6.35.42	Validate pages reachable via static links	Reachable	Heavy	Deterministic
6.35.43	Validate no infinite scroll without pagination	Safe	Heavy	Deterministic
6.35.44	Validate JS navigation crawlable	Crawlable	Heavy	Deterministic
6.35.45	Validate faceted navigation not creating infinite URLs	Safe	Medium	Deterministic
6.35.46	Validate URL parameters sanitized	Clean	Medium	Deterministic
6.35.47	Validate no dynamically generated sessionid URLs in sitemap	None	Medium	Deterministic
6.35.48	Validate rendered HTML matches raw HTML	Matched	Medium	Deterministic
6.35.49	Validate hydration not preventing crawlers from reading content	Accessible	Medium	Deterministic
6.35.50	Compute partial Technical SEO score for Chunk 1	Complete	Critical	Deterministic

URL Structure — Cleanliness, Parameters, Consistency

ID	Check	Passes when	Weight	Runs
6.35.51	Validate URLs clean, readable, hyphen‑separated	Clean	Heavy	Deterministic
6.35.52	Validate no uppercase letters in URLs	Lowercase	Heavy	Deterministic
6.35.53	Validate no encoded characters (%20, %2F)	Clean	Heavy	Deterministic
6.35.54	Validate no irrelevant parameters (utm, ref) indexable	Safe	Medium	Deterministic
6.35.55	Validate URL slugs match page topic	Matched	Medium	Deterministic
6.35.56	Validate trailing slash strategy consistent	Consistent	Medium	Deterministic
6.35.57	Validate URL hierarchy logical	Logical	Medium	Deterministic
6.35.58	Validate no duplicate URLs resolving same content	None	Medium	Deterministic
6.35.59	Validate pagination URLs structured (page/2/)	Structured	Medium	Deterministic
6.35.60	Validate each URL has 1 canonical meaning	Singular	Medium	Deterministic

Duplicate Handling — Canonicalization, Merging, Pruning

ID	Check	Passes when	Weight	Runs
6.35.61	Validate canonical points to preferred version	Correct	Critical	Deterministic
6.35.62	Validate no duplicate indexable content	None	Heavy	Deterministic
6.35.63	Validate HTTP/HTTPS duplicates consolidated	Consolidated	Heavy	Deterministic
6.35.64	Validate www/non‑www duplicates consolidated	Consolidated	Heavy	Deterministic
6.35.65	Validate session IDs stripped from URLs	Stripped	Medium	Deterministic
6.35.66	Validate print‑view or AMP or test pages not indexable	Protected	Medium	Deterministic
6.35.67	Validate canonical not pointing to paginated pages	Clean	Medium	Deterministic
6.35.68	Validate query parameter duplicates handled	Managed	Medium	Deterministic
6.35.69	Validate faceted pages noindex unless curated	Correct	Medium	Deterministic
6.35.70	Validate URL merges/redirects documented	Documented	Medium	Deterministic

Rendered HTML — SSR, CSR, Hydration, Shadow Dom

ID	Check	Passes when	Weight	Runs
6.35.71	Validate page content appears in initial HTML	Present	Heavy	Deterministic
6.35.72	Validate JS-rendered elements accessible in <noscript> fallback	Accessible	Heavy	Deterministic
6.35.73	Validate CSR apps provide SSR/ISR for crawlers	Provided	Heavy	Deterministic
6.35.74	Validate hydration not delaying content availability	Fast	Medium	Deterministic
6.35.75	Validate shadow DOM elements crawled when needed	Crawlable	Medium	Deterministic
6.35.76	Validate React/Vue content not blank prehydrate	Visible	Medium	Deterministic
6.35.77	Validate dynamic components predictable for crawlers	Predictable	Medium	Deterministic
6.35.78	Validate actual text present without JS execution	Present	Medium	Deterministic
6.35.79	Validate lazy‑loaded components not hiding core content	Visible	Medium	Deterministic
6.35.80	Validate SSR & CSR output match content meaning	Matched	Medium	Deterministic

Crawl Budget Optimization — Pruning, Value Prioritization

ID	Check	Passes when	Weight	Runs
6.35.81	Validate no infinite categories/tags generating thousands of pages	Controlled	Critical	Deterministic
6.35.82	Validate low‑value pages marked noindex	Noindex	Heavy	Deterministic
6.35.83	Validate paginated archives not bloating crawl	Minimal	Heavy	Deterministic
6.35.84	Validate search result pages noindexed	Noindex	Heavy	Deterministic
6.35.85	Validate tag pages used only when necessary	Necessary	Medium	Deterministic
6.35.86	Validate parameter pages not included in sitemap	Clean	Medium	Deterministic
6.35.87	Validate “filter results” not indexable	Protected	Medium	Deterministic
6.35.88	Validate archive pages pruned if thin	Pruned	Medium	Deterministic
6.35.89	Validate new pages discoverable within 1 click via hubs	Discoverable	Medium	Deterministic
6.35.90	Validate crawl budget not wasted on parameter permutations	Efficient	Medium	Deterministic

Internal Link Structure — Graph Shape, Depth, Signal Flow

ID	Check	Passes when	Weight	Runs
6.35.91	Validate internal link graph supports crawl depth <4	Efficient	Heavy	Deterministic
6.35.92	Validate hubs link to spokes consistently	Consistent	Heavy	Deterministic
6.35.93	Validate spokes link back to hubs	Return	Heavy	Deterministic
6.35.94	Validate no orphan pages	None	Medium	Deterministic
6.35.95	Validate important URLs receive enough internal links	Sufficient	Medium	Deterministic
6.35.96	Validate link anchors descriptive	Descriptive	Medium	Deterministic
6.35.97	Validate no excessive links in navigation bars	Balanced	Medium	Deterministic
6.35.98	Validate footer links not overwhelming	Balanced	Medium	Deterministic
6.35.99	Validate contextual links present inside content	Present	Medium	Deterministic
6.35.100	Compute partial Technical SEO score for Chunk 2	Complete	Critical	Deterministic

Render Performance — Speed, Lighthouse, AI-Crawler Friendliness

ID	Check	Passes when	Weight	Runs
6.35.101	Validate FCP, LCP, CLS within recommended thresholds	Optimal	Critical	Deterministic
6.35.102	Validate server response time <200ms	Fast	Heavy	Deterministic
6.35.103	Validate image optimization with WebP/AVIF	Optimized	Heavy	Deterministic
6.35.104	Validate critical CSS inlined	Inlined	Heavy	Deterministic
6.35.105	Validate unused CSS removed	Clean	Medium	Deterministic
6.35.106	Validate JS bundles minimized	Minimized	Medium	Deterministic
6.35.107	Validate no render‑blocking JS	None	Medium	Deterministic
6.35.108	Validate lazy‑loading implemented correctly	Correct	Medium	Deterministic
6.35.109	Validate AI‑crawlers can load page without JS	Accessible	Medium	Deterministic
6.35.110	Validate preconnect/dns‑prefetch in place	Present	Medium	Deterministic

Redirect Management — Chains, Loops, Clean Signals

ID	Check	Passes when	Weight	Runs
6.35.111	Validate no redirect loops	None	Critical	Deterministic
6.35.112	Validate redirect chain length ≤2	≤2	Heavy	Deterministic
6.35.113	Validate no mixed redirect signals (301/302 conflict)	None	Heavy	Deterministic
6.35.114	Validate redirect map documented	Documented	Medium	Deterministic
6.35.115	Validate old URLs redirect to exact canonical target	Correct	Medium	Deterministic
6.35.116	Validate no chain from HTTP → HTTPS → WWW → canonical	Single hop	Medium	Deterministic
6.35.117	Validate redirect rules do not break assets	Safe	Medium	Deterministic
6.35.118	Validate locale redirects correct	Correct	Medium	Deterministic
6.35.119	Validate mobile/desktop redirects aligned	Aligned	Medium	Deterministic
6.35.120	Validate no legacy CMS URLs indexable	Protected	Medium	Deterministic

AI-Crawler Compatibility — Stability, Semantic Integrity

ID	Check	Passes when	Weight	Runs
6.35.121	Validate HTML deterministic across requests	Deterministic	Critical	Deterministic
6.35.122	Validate no random content variation on refresh	Stable	Heavy	Deterministic
6.35.123	Validate no time‑based variance in HTML	Stable	Heavy	Deterministic
6.35.124	Validate AI‑agents can parse HTML tree	Parseable	Heavy	Deterministic
6.35.125	Validate <main> tag wraps core content	Correct	Medium	Deterministic
6.35.126	Validate headings predictable for AI interpretation	Predictable	Medium	Deterministic
6.35.127	Validate structured data stable and consistent	Stable	Medium	Deterministic
6.35.128	Validate HTML attribute ordering consistent	Consistent	Medium	Deterministic
6.35.129	Validate whitespace/casing does not break parsers	Safe	Medium	Deterministic
6.35.130	Validate server rendering not personalized	Neutral	Medium	Deterministic

Faceted Navigation — Safe Filters, Pruning, No Infinite URL Space

ID	Check	Passes when	Weight	Runs
6.35.131	Validate no infinite parameter‑generated URLs	None	Critical	Deterministic
6.35.132	Validate filters append safe parameters	Safe	Heavy	Deterministic
6.35.133	Validate filtered URLs noindex unless curated	Noindex	Heavy	Deterministic
6.35.134	Validate parameters order‑agnostic	Standardized	Heavy	Deterministic
6.35.135	Validate category → filter → product path stable	Stable	Medium	Deterministic
6.35.136	Validate price/size/color filters not indexable	Protected	Medium	Deterministic
6.35.137	Validate multi‑filter combinations not crawlable	Protected	Medium	Deterministic
6.35.138	Validate sort parameters neutralized (noindex)	Neutralized	Medium	Deterministic
6.35.139	Validate pagination safe when filtered	Safe	Medium	Deterministic
6.35.140	Validate faceted pages excluded from sitemap	Excluded	Medium	Deterministic

Indexation Health — Search Console, Logs, Coverage

ID	Check	Passes when	Weight	Runs
6.35.141	Validate Search Console Coverage errors absent	None	Critical	Deterministic
6.35.142	Validate submitted vs indexed URLs aligned	Aligned	Heavy	Deterministic
6.35.143	Validate crawled‑but‑not‑indexed low	Low	Heavy	Deterministic
6.35.144	Validate duplicate without user‑selected canonical low	Low	Heavy	Deterministic
6.35.145	Validate excluded URLs intentional	Intentional	Medium	Deterministic
6.35.146	Validate page discovered via sitemaps	Discoverable	Medium	Deterministic
6.35.147	Validate logs show even distribution of crawler hits	Even	Medium	Deterministic
6.35.148	Validate no access‑denied logs for indexable pages	None	Medium	Deterministic
6.35.149	Validate no drastic index volatility	Stable	Medium	Deterministic
6.35.150	Compute partial Technical SEO score for Chunk 3	Complete	Critical	Deterministic

Server & Hosting Configuration — Security, Headers, Protocols

ID	Check	Passes when	Weight	Runs
6.35.151	Validate HTTPS enforced sitewide	Enforced	Critical	Deterministic
6.35.152	Validate HSTS header present	Present	Critical	Deterministic
6.35.153	Validate HTTP → HTTPS redirect single‑hop	Single	Heavy	Deterministic
6.35.154	Validate TLS version ≥1.2	Secure	Heavy	Deterministic
6.35.155	Validate GZip/Brotli compression enabled	Enabled	Medium	Deterministic
6.35.156	Validate security headers (X‑Frame, X‑Content‑Type)	Present	Medium	Deterministic
6.35.157	Validate no mixed content	None	Medium	Deterministic
6.35.158	Validate CDN caching rules correct	Correct	Medium	Deterministic
6.35.159	Validate server compression ratio efficient	Efficient	Medium	Deterministic
6.35.160	Validate HTTP keep‑alive enabled	Enabled	Medium	Deterministic

International Signaling — Hreflang, X-Default, GEO-Targeting

ID	Check	Passes when	Weight	Runs
6.35.161	Validate all language versions linked via hreflang	Linked	Heavy	Deterministic
6.35.162	Validate all hreflang URLs return 200	200	Heavy	Deterministic
6.35.163	Validate country‑specific pages mapped correctly	Correct	Heavy	Deterministic
6.35.164	Validate x‑default exists for global pages	Present	Medium	Deterministic
6.35.165	Validate hreflang reciprocal links	Reciprocal	Medium	Deterministic
6.35.166	Validate no conflicting region vs language tags	Clean	Medium	Deterministic
6.35.167	Validate international sitemaps mapped	Mapped	Medium	Deterministic
6.35.168	Validate geo‑targeting aligns with URLs	Aligned	Medium	Deterministic
6.35.169	Validate hreflang cluster completeness	Complete	Medium	Deterministic
6.35.170	Validate no hreflang errors in Search Console	None	Medium	Deterministic

Site-Wide HTML Sanity — Consistency, Validation, Structure

ID	Check	Passes when	Weight	Runs
6.35.171	Validate HTML passes W3C validation	Valid	Heavy	Deterministic
6.35.172	Validate <title> present on every page	Present	Heavy	Deterministic
6.35.173	Validate <meta charset> correct (UTF‑8)	Correct	Heavy	Deterministic
6.35.174	Validate <html lang=""> correct	Correct	Medium	Deterministic
6.35.175	Validate no duplicated <main> tags	None	Medium	Deterministic
6.35.176	Validate ARIA roles used correctly	Correct	Medium	Deterministic
6.35.177	Validate semantic tags (<article>, <section>) consistent	Consistent	Medium	Deterministic
6.35.178	Validate no malformed HTML blocks	Clean	Medium	Deterministic
6.35.179	Validate no duplicate IDs	None	Medium	Deterministic
6.35.180	Validate iframes sandboxed	Safe	Medium	Deterministic

Advanced Crawlability — Log Analysis, Behavior, Bot Patterns

ID	Check	Passes when	Weight	Runs
6.35.181	Validate logs show consistent crawl behavior	Consistent	Critical	Deterministic
6.35.182	Validate crawl spikes not caused by duplicate traps	Clean	Heavy	Deterministic
6.35.183	Validate indexable pages receiving sufficient crawl hits	Sufficient	Heavy	Deterministic
6.35.184	Validate anomalies in bot activity identified	Identified	Medium	Deterministic
6.35.185	Validate crawl-delay not required	Not required	Medium	Deterministic
6.35.186	Validate no large dead zones in site crawl	None	Medium	Deterministic
6.35.187	Validate JS‑heavy pages crawled efficiently	Efficient	Medium	Deterministic
6.35.188	Validate sitemaps discovered via logs	Discovered	Medium	Deterministic
6.35.189	Validate crawl frequency aligned with content updates	Aligned	Medium	Deterministic
6.35.190	Validate no blocked paths hit frequently	Controlled	Medium	Deterministic

Full Technical SEO Validation — Holistic Check & Final Scoring

ID	Check	Passes when	Weight	Runs
6.35.191	Validate no major crawlability blockers	None	Critical	Deterministic
6.35.192	Validate no major indexability blockers	None	Critical	Deterministic
6.35.193	Validate consistent signals across robots + sitemaps + canonicals	Consistent	Heavy	Deterministic
6.35.194	Validate server behavior predictable for crawlers	Predictable	Heavy	Deterministic
6.35.195	Validate URL structure stable across sessions	Stable	Heavy	Deterministic
6.35.196	Validate technical signals match structured data	Matched	Medium	Deterministic
6.35.197	Validate page discoverability high	High	Medium	Deterministic
6.35.198	Validate dynamic pages treated safely	Safe	Medium	Deterministic
6.35.199	Identify remaining technical gaps	Identified	Medium	Deterministic
6.35.200	Compute full TECHNICAL SEO composite score	Finalized	Critical	Deterministic

← 6.34 Content Quality / E-E-A-T / Thematic Authority / NLP-Semantics / AI-Readability & Favorability Signal Suite6.36 Core Web Vitals / Speed Optimization / Performance AI-Readiness →

Run these checks on your site

A Deep Audit scores every check here that applies to your page, then an AI pass of up to 150 checks. Included on Pro and Ultra, or $9 for one audit.

See plans

Machine-readable: catalog totals and the full catalog as JSON (IDs, section, weight, status).