Home / Check catalog / 6. Schema, Local, Trust, Deep Audit / 6.39 Technical SEO (Crawlability, Indexation, Sitemaps, Canonicals, Duplication)

6.39 Technical SEO (Crawlability, Indexation, Sitemaps, Canonicals, Duplication)

200 checks in Schema, Local, Trust, Deep Audit. Automated Its subsection has a live automated checker in the engine.

robots.txt — Rules, Safety, Blocking Logic

ID	Check	Passes when	Weight	Runs
6.39.01	Validate robots.txt exists	Present	Critical	Deterministic
6.39.02	Validate robots.txt publicly accessible	Accessible	Critical	Deterministic
6.39.03	Validate no accidental global Disallow: /	Safe	Critical	Deterministic
6.39.04	Validate essential pages NOT disallowed	Allowed	Heavy	Deterministic
6.39.05	Validate disallow rules correctly formatted	Correct	Heavy	Deterministic
6.39.06	Validate wildcards (*) used properly	Correct	Heavy	Deterministic
6.39.07	Validate crawl-delay not excessively high	Balanced	Medium	Deterministic
6.39.08	Validate sitemap listed in robots.txt	Listed	Medium	Deterministic
6.39.09	Validate blocked paths intentional	Intentional	Medium	Deterministic
6.39.10	Validate AI-agent access not incorrectly blocked	Accessible	Medium	Deterministic

Crawlability — Fetchability, Renderability, URL Access

ID	Check	Passes when	Weight	Runs
6.39.11	Validate all key URLs return 200 status	200	Critical	Deterministic
6.39.12	Validate no essential pages behind JS-only navigation	Crawlable	Critical	Deterministic
6.39.13	Validate page fully fetchable by Googlebot	Fetchable	Heavy	Deterministic
6.39.14	Validate server not blocking Googlebot IPs	Unblocked	Heavy	Deterministic
6.39.15	Validate infinite scroll has crawlable fallback	Fallback	Heavy	Deterministic
6.39.16	Validate robots meta tags correctly used (index/noindex)	Correct	Medium	Deterministic
6.39.17	Validate navigation links crawlable (no JS traps)	Crawlable	Medium	Deterministic
6.39.18	Validate faceted navigation not infinite	Safe	Medium	Deterministic
6.39.19	Validate pagination crawlable (rel=prev/next if used)	Crawlable	Medium	Deterministic
6.39.20	Validate no blocked CSS/JS that prevents rendering	Accessible	Medium	Deterministic

Indexation — Directives, Signal Consistency, Intent

ID	Check	Passes when	Weight	Runs
6.39.21	Validate all indexable pages include “index” directives implicitly or explicitly	Indexable	Critical	Deterministic
6.39.22	Validate no conflicting index/noindex signals	Non-conflicting	Critical	Deterministic
6.39.23	Validate canonical tag present on all pages	Present	Heavy	Deterministic
6.39.24	Validate canonical tag correct (self-referencing if appropriate)	Correct	Heavy	Deterministic
6.39.25	Validate canonical URL uses https not http	Secure	Heavy	Deterministic
6.39.26	Validate paginated canonical logic correct	Correct	Medium	Deterministic
6.39.27	Validate parameterized URLs canonicalized correctly	Canonicalized	Medium	Deterministic
6.39.28	Validate canonical not pointing to non-existent pages	Valid	Medium	Deterministic
6.39.29	Validate canonical and hreflang consistent	Consistent	Medium	Deterministic
6.39.30	Validate canonical prevents duplicate clusters	Preventive	Medium	Deterministic

Sitemaps — XML, Image, Video, Index, Hreflang

ID	Check	Passes when	Weight	Runs
6.39.31	Validate XML sitemap exists	Present	Heavy	Deterministic
6.39.32	Validate XML sitemap valid (no parsing errors)	Valid	Heavy	Deterministic
6.39.33	Validate <loc> URLs correct & absolute	Correct	Heavy	Deterministic
6.39.34	Validate lastmod present & accurate	Accurate	Medium	Deterministic
6.39.35	Validate image sitemap present if image-heavy site	Present	Medium	Deterministic
6.39.36	Validate video sitemap present if video-heavy	Present	Medium	Deterministic
6.39.37	Validate news sitemap correct (if applicable)	Correct	Medium	Deterministic
6.39.38	Validate hreflang sitemap correct (if multilingual)	Correct	Medium	Deterministic
6.39.39	Validate sitemap index file correct (if site > 50k URLs)	Correct	Medium	Deterministic
6.39.40	Validate sitemap only includes canonical URLs	Canonical	Medium	Deterministic

Duplication — Thin Pages, Near-Dupes, Param Dupes

ID	Check	Passes when	Weight	Runs
6.39.41	Validate no URL parameters create infinite duplicates	Safe	Heavy	Deterministic
6.39.42	Validate duplicate content clusters correctly canonicalized	Canonicalized	Heavy	Deterministic
6.39.43	Validate no doorway pages	None	Heavy	Deterministic
6.39.44	Validate thin pages (<150 words) minimized	Minimal	Medium	Deterministic
6.39.45	Validate boilerplate repetition minimized	Reduced	Medium	Deterministic
6.39.46	Validate trailing slash rules consistent	Consistent	Medium	Deterministic
6.39.47	Validate uppercase/lowercase URL consistency	Consistent	Medium	Deterministic
6.39.48	Validate session IDs not in URLs	Clean	Medium	Deterministic
6.39.49	Validate print URLs blocked from indexation	Blocked	Medium	Deterministic
6.39.50	Compute partial Technical SEO score for Chunk 1	Complete	Critical	Deterministic

Crawl Budget Optimization — Efficiency, Priority, Bot Behavior

ID	Check	Passes when	Weight	Runs
6.39.51	Validate crawl budget not wasted on low‑value URLs	Efficient	Heavy	Deterministic
6.39.52	Validate no infinite URL generation patterns	Safe	Heavy	Deterministic
6.39.53	Validate limited duplicate rendering paths	Limited	Heavy	Deterministic
6.39.54	Validate pagination URLs finite	Finite	Medium	Deterministic
6.39.55	Validate parameters don’t explode crawl paths	Controlled	Medium	Deterministic
6.39.56	Validate server handles crawl spikes	Stable	Medium	Deterministic
6.39.57	Validate XML sitemaps reduce crawl waste	Reduced	Medium	Deterministic
6.39.58	Validate proper “priority” values (if included)	Logical	Medium	Deterministic
6.39.59	Validate “changefreq” values accurate	Accurate	Medium	Deterministic
6.39.60	Validate no orphaned indexable pages	None	Medium	Deterministic

Internal Linking & Structure — Crawl Flow, Bot Paths

ID	Check	Passes when	Weight	Runs
6.39.61	Validate internal links crawlable	Crawlable	Heavy	Deterministic
6.39.62	Validate anchor tags not replaced by JS onclick handlers	Standard	Heavy	Deterministic
6.39.63	Validate all indexable pages reachable in ≤3 clicks	Reachable	Heavy	Deterministic
6.39.64	Validate logical hub‑to‑spoke structure	Structured	Medium	Deterministic
6.39.65	Validate no internal redirects	Clean	Medium	Deterministic
6.39.66	Validate breadcrumb navigation crawlable	Crawlable	Medium	Deterministic
6.39.67	Validate pagination internally linked	Linked	Medium	Deterministic
6.39.68	Validate faceted links safely handled	Safe	Medium	Deterministic
6.39.69	Validate sidebar/footer links not excessive	Balanced	Medium	Deterministic
6.39.70	Validate no hidden link spam	Clean	Medium	Deterministic

HTTP Status & Server Behavior — Reliability, Consistency

ID	Check	Passes when	Weight	Runs
6.39.71	Validate no soft 404s	None	Critical	Deterministic
6.39.72	Validate no server errors (5xx)	None	Critical	Deterministic
6.39.73	Validate 4xx errors minimized	Minimal	Heavy	Deterministic
6.39.74	Validate redirect chains < 2 hops	Short	Heavy	Deterministic
6.39.75	Validate HTTP→HTTPS redirect correct	Correct	Heavy	Deterministic
6.39.76	Validate non‑www→www redirect consistent (or vice versa)	Unified	Medium	Deterministic
6.39.77	Validate no mixed protocol content	Clean	Medium	Deterministic
6.39.78	Validate no redirect loops	None	Medium	Deterministic
6.39.79	Validate 410 used for intentionally removed content	Correct	Medium	Deterministic
6.39.80	Validate HEAD requests return accurate headers	Accurate	Medium	Deterministic

AI Indexability — Predictable Rendering, Clean Dom, Consistent Signals

ID	Check	Passes when	Weight	Runs
6.39.81	Validate HTML DOM deterministic (no random generation)	Deterministic	Critical	Deterministic
6.39.82	Validate page fully understandable without JS execution	Understandable	Critical	Deterministic
6.39.83	Validate structured data available without JS	Available	Heavy	Deterministic
6.39.84	Validate DOM stable during parse (no layout shift heavy JS)	Stable	Heavy	Deterministic
6.39.85	Validate core content visible in initial HTML	Visible	Heavy	Deterministic
6.39.86	Validate React/Next.js hydration not blocking content	Non-blocking	Medium	Deterministic
6.39.87	Validate deterministic URL structure	Deterministic	Medium	Deterministic
6.39.88	Validate no meaningless DOM wrappers	Clean	Medium	Deterministic
6.39.89	Validate no random client-side params in URLs	Stable	Medium	Deterministic
6.39.90	Validate HTML token sequence predictable	Predictable	Medium	Deterministic

Canonical Cluster Integrity — Duplicate Prevention, Consistency

ID	Check	Passes when	Weight	Runs
6.39.91	Validate one canonical per page	Singular	Heavy	Deterministic
6.39.92	Validate canonical points to canonical version	Accurate	Heavy	Deterministic
6.39.93	Validate canonical clusters small & stable	Stable	Heavy	Deterministic
6.39.94	Validate UTM/parameters removed from canonical	Clean	Medium	Deterministic
6.39.95	Validate canonical matches sitemap	Matched	Medium	Deterministic
6.39.96	Validate canonical doesn’t conflict robots rules	Consistent	Medium	Deterministic
6.39.97	Validate canonical chain not circular	Linear	Medium	Deterministic
6.39.98	Validate canonical references https not http	Secure	Medium	Deterministic
6.39.99	Validate hreflang + canonical pairing correct	Correct	Medium	Deterministic
6.39.100	Compute partial Technical SEO score for Chunk 2	Complete	Critical	Deterministic

URL Structure — Cleanliness, Normalization, Predictability

ID	Check	Passes when	Weight	Runs
6.39.101	Validate URLs human‑readable	Clean	Heavy	Deterministic
6.39.102	Validate URLs contain no gibberish strings	Clean	Heavy	Deterministic
6.39.103	Validate hyphens used properly in URLs	Hyphenated	Heavy	Deterministic
6.39.104	Validate lowercase URLs only	Lowercase	Heavy	Deterministic
6.39.105	Validate no trailing slash inconsistencies	Consistent	Medium	Deterministic
6.39.106	Validate no duplicated paths (e.g., /page/page)	Unique	Medium	Deterministic
6.39.107	Validate parameter ordering consistent	Ordered	Medium	Deterministic
6.39.108	Validate category/service URLs follow logical hierarchy	Hierarchical	Medium	Deterministic
6.39.109	Validate no randomly generated slugs	Stable	Medium	Deterministic
6.39.110	Validate URL naming conventions documented	Documented	Medium	Deterministic

Server Headers — X-Robots-Tag, Index Signals, Security

ID	Check	Passes when	Weight	Runs
6.39.111	Validate x‑robots‑tag headers not conflicting	Non-conflicting	Heavy	Deterministic
6.39.112	Validate noindex in headers matches HTML noindex	Matched	Heavy	Deterministic
6.39.113	Validate canonical hints not overridden by headers	Consistent	Heavy	Deterministic
6.39.114	Validate security headers present (HSTS, X‑Frame‑Options)	Present	Medium	Deterministic
6.39.115	Validate cache-control headers logical	Logical	Medium	Deterministic
6.39.116	Validate no-index headers not applied accidentally	Safe	Medium	Deterministic
6.39.117	Validate content-type headers correct	Correct	Medium	Deterministic
6.39.118	Validate language headers consistent with hreflang	Consistent	Medium	Deterministic
6.39.119	Validate compression (gzip/br) enabled	Enabled	Medium	Deterministic
6.39.120	Validate no caching conflicts for HTML	Non-conflicting	Medium	Deterministic

Redirect Logic — Permanence, Accuracy, Clean Mapping

ID	Check	Passes when	Weight	Runs
6.39.121	Validate 301 used for permanent redirects	Correct	Heavy	Deterministic
6.39.122	Validate redirect mapping complete	Complete	Heavy	Deterministic
6.39.123	Validate redirect chains minimal	Minimal	Heavy	Deterministic
6.39.124	Validate internal links updated (not relying on redirect)	Updated	Medium	Deterministic
6.39.125	Validate no 302 used for permanent moves	Corrected	Medium	Deterministic
6.39.126	Validate trailing-slash redirect consistent	Consistent	Medium	Deterministic
6.39.127	Validate uppercase redirects normalized	Normalized	Medium	Deterministic
6.39.128	Validate mobile/AMP redirects correct	Correct	Medium	Deterministic
6.39.129	Validate redirect rules not conflicting	Non-conflicting	Medium	Deterministic
6.39.130	Validate server-side redirect logic documented	Documented	Medium	Deterministic

XML Sitemap Integrity — Deep Validation

ID	Check	Passes when	Weight	Runs
6.39.131	Validate sitemap URLs return 200	200	Heavy	Deterministic
6.39.132	Validate sitemap URLs canonical	Canonical	Heavy	Deterministic
6.39.133	Validate sitemap URLs indexable	Indexable	Heavy	Deterministic
6.39.134	Validate lastmod not fake	Truthful	Medium	Deterministic
6.39.135	Validate sitemap not too large (>50k URLs)	Sized	Medium	Deterministic
6.39.136	Validate sitemap pagination correct	Correct	Medium	Deterministic
6.39.137	Validate image/video sitemap references accurate	Accurate	Medium	Deterministic
6.39.138	Validate hreflang sitemap URLs match canonical	Matched	Medium	Deterministic
6.39.139	Validate no non-canonical in sitemap	Clean	Medium	Deterministic
6.39.140	Validate no 404/500 URLs in sitemap	Clean	Medium	Deterministic

Duplication & Thin Content — Advanced Automated Detection

ID	Check	Passes when	Weight	Runs
6.39.141	Validate near-duplicate sets detected	Detected	Heavy	Deterministic
6.39.142	Validate canonical clusters used for duplicates	Clustered	Heavy	Deterministic
6.39.143	Validate duplicate parameter permutations identified	Identified	Heavy	Deterministic
6.39.144	Validate similar titles/descriptions flagged	Flagged	Medium	Deterministic
6.39.145	Validate templated content not causing mass duplication	Controlled	Medium	Deterministic
6.39.146	Validate duplicate H1/H2 clusters minimized	Minimized	Medium	Deterministic
6.39.147	Validate content uniqueness >70% for indexable pages	Unique	Medium	Deterministic
6.39.148	Validate no thin RSS/Atom/index feed pages indexable	Blocked	Medium	Deterministic
6.39.149	Validate soft duplicate pages flagged	Flagged	Medium	Deterministic
6.39.150	Compute partial Technical SEO score for Chunk 3	Complete	Critical	Deterministic

Advanced Indexation Conflicts — Preventing Mixed Signal Failures

ID	Check	Passes when	Weight	Runs
6.39.151	Validate canonical does not conflict with noindex	Non-conflicting	Heavy	Deterministic
6.39.152	Validate robots.txt does not conflict with canonical target	Aligned	Heavy	Deterministic
6.39.153	Validate canonical target is indexable	Indexable	Heavy	Deterministic
6.39.154	Validate internal links point to canonical URLs	Consistent	Heavy	Deterministic
6.39.155	Validate hreflang references canonical version	Correct	Medium	Deterministic
6.39.156	Validate sitemap contains only indexable URLs	Clean	Medium	Deterministic
6.39.157	Validate meta robots matches x‑robots‑tag header	Matched	Medium	Deterministic
6.39.158	Validate noindex not mistakenly applied to templates	Safe	Medium	Deterministic
6.39.159	Validate duplication handling consistent across languages	Consistent	Medium	Deterministic
6.39.160	Validate canonical chain length = 1	Short	Medium	Deterministic

JavaScript Renderability — SSR, Hydration, Fallbacks

ID	Check	Passes when	Weight	Runs
6.39.161	Validate essential content visible without JS	Visible	Critical	Deterministic
6.39.162	Validate SSR or static fallback exists	Exists	Heavy	Deterministic
6.39.163	Validate hydration doesn't overwrite pre-rendered content	Safe	Heavy	Deterministic
6.39.164	Validate router-based navigation crawlable	Crawlable	Heavy	Deterministic
6.39.165	Validate no JS-generated duplicate URLs	Controlled	Medium	Deterministic
6.39.166	Validate onClick navigation backed by actual <a> tag	Standards	Medium	Deterministic
6.39.167	Validate no hydration mismatches	Stable	Medium	Deterministic
6.39.168	Validate JS bundles minimized	Minimal	Medium	Deterministic
6.39.169	Validate no client-side URL randomization	Deterministic	Medium	Deterministic
6.39.170	Validate DOM not mutated excessively post-render	Stable	Medium	Deterministic

Crawl Traps — Detecting & Preventing Bot Dead-Ends

ID	Check	Passes when	Weight	Runs
6.39.171	Validate no infinite calendar URLs	None	Heavy	Deterministic
6.39.172	Validate no infinite faceted filters	Controlled	Heavy	Deterministic
6.39.173	Validate no infinite “?page=999” URLs	Limited	Heavy	Deterministic
6.39.174	Validate no dynamically appended hashes (#) create duplicates	Clean	Medium	Deterministic
6.39.175	Validate no sorting parameters multiply pages	Safe	Medium	Deterministic
6.39.176	Validate search URLs blocked or controlled	Controlled	Medium	Deterministic
6.39.177	Validate filter permutations finite	Finite	Medium	Deterministic
6.39.178	Validate session-based URLs not crawlable	Blocked	Medium	Deterministic
6.39.179	Validate calendar/archive URLs correctly limited	Limited	Medium	Deterministic
6.39.180	Validate print/share URLs non-indexable	Blocked	Medium	Deterministic

Hreflang System — Cross-Language Canonical, Region Logic

ID	Check	Passes when	Weight	Runs
6.39.181	Validate hreflang tags present for multilingual sites	Present	Heavy	Deterministic
6.39.182	Validate hreflang links reciprocal	Reciprocal	Heavy	Deterministic
6.39.183	Validate hreflang targets canonical version	Canonical	Heavy	Deterministic
6.39.184	Validate regional codes correct (en-US vs en-GB)	Correct	Medium	Deterministic
6.39.185	Validate no mismatched canonicals between languages	Consistent	Medium	Deterministic
6.39.186	Validate language alternates complete	Complete	Medium	Deterministic
6.39.187	Validate hreflang clusters fully connected	Connected	Medium	Deterministic
6.39.188	Validate country-targeting logical	Logical	Medium	Deterministic
6.39.189	Validate hreflang error-free	Valid	Medium	Deterministic
6.39.190	Validate sitemap includes hreflang (if using sitemap method)	Included	Medium	Deterministic

Final Technical SEO Validation — Holistic Indexation State

ID	Check	Passes when	Weight	Runs
6.39.191	Validate no contradictory indexation signals remain	Clean	Critical	Deterministic
6.39.192	Validate crawlability of entire site map	Fully crawlable	Critical	Deterministic
6.39.193	Validate canonical system coherent	Coherent	Heavy	Deterministic
6.39.194	Validate structured data discoverable for all key pages	Discoverable	Heavy	Deterministic
6.39.195	Validate crawl budget optimized	Optimized	Heavy	Deterministic
6.39.196	Validate index coverage healthy (no sharp drops)	Healthy	Medium	Deterministic
6.39.197	Validate no soft duplicates left unhandled	Clean	Medium	Deterministic
6.39.198	Validate all sitemaps referenced in robots.txt	Referenced	Medium	Deterministic
6.39.199	Identify any remaining indexation gaps	Identified	Medium	Deterministic
6.39.200	Compute FULL TECHNICAL SEO composite score	Finalized	Critical	Deterministic

← 6.38 Multimedia SEO (Images / Video / Audio) + AI Multimodal Readiness6.40 Local SEO (Hybrid Model) →

Run these checks on your site

A Deep Audit scores every check here that applies to your page, then an AI pass of up to 150 checks. Included on Pro and Ultra, or $9 for one audit.

See plans

Machine-readable: catalog totals and the full catalog as JSON (IDs, section, weight, status).