Home / Check catalog / 6. Schema, Local, Trust, Deep Audit / 6.43 Technical SEO, Crawling & Indexability

6.43 Technical SEO, Crawling & Indexability

200 checks in Schema, Local, Trust, Deep Audit. Automated Its subsection has a live automated checker in the engine.

robots.txt Validation

ID	Check	Passes when	Weight	Runs
6.43.01	Detect robots.txt file presence	Present	Critical	Deterministic
6.43.02	Validate robots.txt syntax correctness	Valid	Critical	Deterministic
6.43.03	Validate no accidental Disallow: / applied globally	Clean	Critical	Deterministic
6.43.04	Validate Googlebot allowed for core pages	Allowed	Heavy	Deterministic
6.43.05	Validate AI crawlers (GPTBot, CCBot, PerplexityBot) configured correctly	Correct	Heavy	Deterministic
6.43.06	Validate media bots allowed (images, videos)	Allowed	Heavy	Deterministic
6.43.07	Validate sitemap directive included	Included	Medium	Deterministic
6.43.08	Detect malformed or unused wildcard patterns	None	Medium	Deterministic
6.43.09	Validate crawl-delay not misused	Correct	Medium	Deterministic
6.43.10	Detect obsolete directives (e.g., noindex in robots.txt)	None	Medium	Deterministic

Sitemap Infrastructure (XML + Index Sitemaps)

ID	Check	Passes when	Weight	Runs
6.43.11	Detect sitemap.xml presence	Present	Critical	Deterministic
6.43.12	Validate XML sitemap accessibility	Accessible	Critical	Deterministic
6.43.13	Validate sitemap index structure (if multiple sitemaps exist)	Correct	Heavy	Deterministic
6.43.14	Validate all sitemap entries return 200 OK	OK	Heavy	Deterministic
6.43.15	Validate sitemap lastmod accurate for each URL	Accurate	Heavy	Deterministic
6.43.16	Validate image/video sitemap correctness (if exists)	Correct	Medium	Deterministic
6.43.17	Detect orphaned URLs in sitemap	None	Medium	Deterministic
6.43.18	Validate sitemap does not include noindex URLs	Clean	Medium	Deterministic
6.43.19	Validate sitemap size within limits	Within limits	Medium	Deterministic
6.43.20	Detect redirects inside sitemap	None	Medium	Deterministic

Core Crawlability Checks

ID	Check	Passes when	Weight	Runs
6.43.21	Validate homepage crawlable	Crawlable	Critical	Deterministic
6.43.22	Validate top-level pages crawlable (About, Services, Contact, etc.)	Crawlable	Critical	Deterministic
6.43.23	Validate service/location pages crawlable	Crawlable	Heavy	Deterministic
6.43.24	Detect blocked core directories	None	Heavy	Deterministic
6.43.25	Validate JS/CSS not disallowed in robots.txt	Allowed	Heavy	Deterministic
6.43.26	Validate correct internal linking for discoverability	Correct	Medium	Deterministic
6.43.27	Detect pages requiring login for crawler access	None	Medium	Deterministic
6.43.28	Validate infinite-scroll pages handled	Handled	Medium	Deterministic
6.43.29	Validate faceted navigation crawl risk minimized	Minimized	Medium	Deterministic
6.43.30	Validate AJAX pages have crawlable equivalents	Provided	Medium	Deterministic

URL & Path Accessibility

ID	Check	Passes when	Weight	Runs
6.43.31	Validate URL paths clean and static	Clean	Heavy	Deterministic
6.43.32	Validate no session IDs in URLs	None	Heavy	Deterministic
6.43.33	Validate no uppercase/lowercase duplicates	Clean	Heavy	Deterministic
6.43.34	Validate hyphens used correctly (not underscores)	Correct	Medium	Deterministic
6.43.35	Validate URLs short and descriptive	Descriptive	Medium	Deterministic
6.43.36	Detect query-parameter URL duplicates	None	Medium	Deterministic
6.43.37	Validate correct separation between categories/subdirectories	Correct	Medium	Deterministic
6.43.38	Validate pagination URLs consistent	Consistent	Medium	Deterministic
6.43.39	Detect trailing-slash inconsistency	Consistent	Medium	Deterministic
6.43.40	Validate lowercase enforced globally	Enforced	Medium	Deterministic

Mobile-First Crawling

ID	Check	Passes when	Weight	Runs
6.43.41	Validate site mobile‑first crawlable	Passed	Critical	Deterministic
6.43.42	Validate mobile version identical in content	Identical	Heavy	Deterministic
6.43.43	Validate mobile resources loaded without blocking	Loaded	Heavy	Deterministic
6.43.44	Validate responsive design meta tag correct	Correct	Medium	Deterministic
6.43.45	Validate mobile UX elements (menus, buttons) crawlable	Crawlable	Medium	Deterministic
6.43.46	Detect mobile-only hidden content	None	Medium	Deterministic
6.43.47	Validate mobile images sized correctly	Correct	Medium	Deterministic
6.43.48	Detect mobile resource blocking (CSS/JS)	None	Medium	Deterministic
6.43.49	Validate no intrusive interstitials	None	Medium	Deterministic
6.43.50	Compute partial Crawlability score	Finalized	Critical	Deterministic

Indexability Validation

ID	Check	Passes when	Weight	Runs
6.43.51	Detect indexability of homepage	Indexable	Critical	Deterministic
6.43.52	Validate essential pages return indexable signals	Indexable	Critical	Deterministic
6.43.53	Validate <meta name="robots"> tags correct	Correct	Heavy	Deterministic
6.43.54	Detect accidental noindex tags	None	Heavy	Deterministic
6.43.55	Validate HTTP headers do not override indexability	Clean	Heavy	Deterministic
6.43.56	Validate proper placement of noindex on thin pages	Correct	Medium	Deterministic
6.43.57	Detect index bloat from duplicate content	None	Medium	Deterministic
6.43.58	Validate pagination indexation rules correct	Correct	Medium	Deterministic
6.43.59	Detect infinite URL loops	None	Medium	Deterministic
6.43.60	Validate hreflang-supported pages indexable	Indexable	Medium	Deterministic

Canonical Tags & Duplication Control

ID	Check	Passes when	Weight	Runs
6.43.61	Validate canonical tag present	Present	Critical	Deterministic
6.43.62	Validate canonical tag self-referencing (when appropriate)	Self-referencing	Critical	Deterministic
6.43.63	Validate canonical chain not broken	Clean	Heavy	Deterministic
6.43.64	Validate no conflicting canonical + meta robots combinations	None	Heavy	Deterministic
6.43.65	Validate canonical URLs match primary content version	Matching	Heavy	Deterministic
6.43.66	Detect duplicate canonical URLs across unrelated pages	None	Medium	Deterministic
6.43.67	Validate canonicalization of parameters	Correct	Medium	Deterministic
6.43.68	Detect self-canonical mismatch	None	Medium	Deterministic
6.43.69	Validate canonical tags use absolute URLs	Absolute	Medium	Deterministic
6.43.70	Validate consistency between sitemap URL and canonical URL	Consistent	Medium	Deterministic

Content Duplication & Language Variants

ID	Check	Passes when	Weight	Runs
6.43.71	Detect full-page duplication	None	Heavy	Deterministic
6.43.72	Detect near-duplicate content clusters	None	Heavy	Deterministic
6.43.73	Validate unique titles and descriptions per page	Unique	Heavy	Deterministic
6.43.74	Validate duplicate H1 structures eliminated	Clean	Medium	Deterministic
6.43.75	Validate language variants not duplicating each other	Unique	Medium	Deterministic
6.43.76	Validate hreflang pairs consistent	Consistent	Medium	Deterministic
6.43.77	Detect hreflang return-tag mismatches	None	Medium	Deterministic
6.43.78	Validate each region/language has correct canonical	Correct	Medium	Deterministic
6.43.79	Detect accidental mixed-language pages	None	Medium	Deterministic
6.43.80	Validate language declaration matches actual content	Matching	Medium	Deterministic

Rendering Validation (JS, SSR, Hydration)

ID	Check	Passes when	Weight	Runs
6.43.81	Validate page renderable by Googlebot (HTML snapshot)	Renderable	Critical	Deterministic
6.43.82	Validate JavaScript not blocking initial content rendering	Non-blocking	Heavy	Deterministic
6.43.83	Validate dynamic content appears in rendered HTML	Present	Heavy	Deterministic
6.43.84	Validate hydration errors not present	None	Heavy	Deterministic
6.43.85	Validate lazy-loaded elements rendered after JS execution	Rendered	Medium	Deterministic
6.43.86	Validate no critical content rendered only client-side	None	Medium	Deterministic
6.43.87	Validate script tags follow async/defer best practices	Correct	Medium	Deterministic
6.43.88	Validate inline JS minimized	Minimal	Medium	Deterministic
6.43.89	Validate JS errors not preventing crawler logic	Clean	Medium	Deterministic
6.43.90	Detect shadow-DOM hidden content	None	Medium	Deterministic

Core Content Checks After Rendering

ID	Check	Passes when	Weight	Runs
6.43.91	Validate rendered HTML contains correct H1	Present	Heavy	Deterministic
6.43.92	Validate rendered meta tags identical to pre-render	Matching	Heavy	Deterministic
6.43.93	Validate structured data present after JS render	Present	Heavy	Deterministic
6.43.94	Validate navigation links render properly	Rendered	Medium	Deterministic
6.43.95	Validate footer elements render properly	Rendered	Medium	Deterministic
6.43.96	Validate mobile menu renders after JS execution	Functional	Medium	Deterministic
6.43.97	Validate JS-rendered reviews/testimonials visible	Visible	Medium	Deterministic
6.43.98	Validate dynamic forms visible (contact, checkout, lead forms)	Visible	Medium	Deterministic
6.43.99	Detect content lost between raw HTML and rendered DOM	None	Medium	Deterministic
6.43.100	Compute partial Indexability & Rendering score	Finalized	Critical	Deterministic

Server Response Integrity

ID	Check	Passes when	Weight	Runs
6.43.101	Validate homepage returns 200 OK	OK	Critical	Deterministic
6.43.102	Validate all major pages return 200 OK	OK	Critical	Deterministic
6.43.103	Validate no soft‑404s	None	Heavy	Deterministic
6.43.104	Detect 5xx server errors across site	None	Heavy	Deterministic
6.43.105	Detect 4xx errors on internal links	None	Heavy	Deterministic
6.43.106	Validate server response times healthy (<600ms)	Healthy	Medium	Deterministic
6.43.107	Validate correct caching headers present	Present	Medium	Deterministic
6.43.108	Detect inconsistent MIME types	None	Medium	Deterministic
6.43.109	Validate GZIP/Brotli compression enabled	Enabled	Medium	Deterministic
6.43.110	Validate no mixed‑content warnings	Clean	Medium	Deterministic

Redirects & Canonical Path Control

ID	Check	Passes when	Weight	Runs
6.43.111	Validate WWW → non‑WWW (or vice‑versa) enforced	Enforced	Critical	Deterministic
6.43.112	Validate HTTP → HTTPS redirect	Enforced	Critical	Deterministic
6.43.113	Detect redirect chains (>1 hop)	Minimal	Heavy	Deterministic
6.43.114	Detect redirect loops	None	Heavy	Deterministic
6.43.115	Validate canonical redirect logic correct	Correct	Heavy	Deterministic
6.43.116	Validate legacy URLs redirect cleanly to canonical URLs	Clean	Medium	Deterministic
6.43.117	Validate 301 used for permanent redirection	Correct	Medium	Deterministic
6.43.118	Validate 302/307 used only for temporary flows	Correct	Medium	Deterministic
6.43.119	Detect meta refresh redirects	None	Medium	Deterministic
6.43.120	Detect JavaScript redirects	None	Medium	Deterministic

Core Infrastructure (DNS, Hosting, SSL)

ID	Check	Passes when	Weight	Runs
6.43.121	Validate DNS resolves correctly across global regions	Resolved	Critical	Deterministic
6.43.122	Validate correct DNS records (A, AAAA, CNAME, MX)	Correct	Heavy	Deterministic
6.43.123	Validate no duplicate A/AAAA records	Clean	Heavy	Deterministic
6.43.124	Validate DNS TTL reasonable (300–3600 sec)	Reasonable	Medium	Deterministic
6.43.125	Validate authoritative nameservers correct	Correct	Medium	Deterministic
6.43.126	Detect slow DNS lookup times	None	Medium	Deterministic
6.43.127	Validate SSL certificate issuer reputable	Reputable	Medium	Deterministic
6.43.128	Validate certificate expiration >30 days	Valid	Medium	Deterministic
6.43.129	Validate certificate chain correctly configured	Correct	Medium	Deterministic
6.43.130	Detect insecure legacy ciphers	None	Medium	Deterministic

Resource Integrity & Loading Behavior

ID	Check	Passes when	Weight	Runs
6.43.131	Validate JS/CSS files return 200 OK	OK	Heavy	Deterministic
6.43.132	Detect unused CSS	Minimal	Heavy	Deterministic
6.43.133	Detect unused JS	Minimal	Heavy	Deterministic
6.43.134	Validate resource minification (JS/CSS)	Minified	Medium	Deterministic
6.43.135	Validate preload/preconnect hints correct	Correct	Medium	Deterministic
6.43.136	Validate image lazy‑loading correctly implemented	Correct	Medium	Deterministic
6.43.137	Detect render‑blocking CSS/JS	None	Medium	Deterministic
6.43.138	Validate fonts loaded efficiently	Efficient	Medium	Deterministic
6.43.139	Validate cookie & tracking scripts not blocking LCP	Clean	Medium	Deterministic
6.43.140	Detect unused preflight CORS requests	None	Medium	Deterministic

Hosting & Server Architecture

ID	Check	Passes when	Weight	Runs
6.43.141	Detect shared hosting performance limitations	Minimal	Heavy	Deterministic
6.43.142	Validate CDN usage	Used	Heavy	Deterministic
6.43.143	Validate CDN edge cache hit rate healthy	Healthy	Heavy	Deterministic
6.43.144	Validate CDN correctly configured (headers, compression)	Correct	Medium	Deterministic
6.43.145	Validate hosting geolocation aligned with target users	Aligned	Medium	Deterministic
6.43.146	Detect hosting IP historical blacklist issues	None	Medium	Deterministic
6.43.147	Validate server OS up‑to‑date	Updated	Medium	Deterministic
6.43.148	Detect outdated Apache/Nginx versions	None	Medium	Deterministic
6.43.149	Validate hosting uptime >99.9%	High	Medium	Deterministic
6.43.150	Compute partial Server & Redirect Integrity score	Finalized	Critical	Deterministic

Core Web Vitals (LCP, FID/INP, CLS)

ID	Check	Passes when	Weight	Runs
6.43.151	Validate LCP element identifiable	Identified	Critical	Deterministic
6.43.152	Validate LCP < 2.5s (mobile)	Passed	Critical	Deterministic
6.43.153	Validate INP (formerly FID) < 200ms	Passed	Heavy	Deterministic
6.43.154	Validate CLS < 0.1	Passed	Heavy	Deterministic
6.43.155	Detect layout shifts from images without dimensions	None	Heavy	Deterministic
6.43.156	Validate preloading of LCP resource	Preloaded	Medium	Deterministic
6.43.157	Detect render-blocking CSS affecting LCP	None	Medium	Deterministic
6.43.158	Validate font-display swap for faster rendering	Enabled	Medium	Deterministic
6.43.159	Validate JS execution time not inflating INP	Controlled	Medium	Deterministic
6.43.160	Validate no sudden CLS from ads, embeds	None	Medium	Deterministic

Page Performance (Lighthouse + Real-User Data)

ID	Check	Passes when	Weight	Runs
6.43.161	Validate Time to First Byte (TTFB) healthy (<500ms)	Healthy	Heavy	Deterministic
6.43.162	Validate First Contentful Paint (FCP) < 1.8s	Passed	Heavy	Deterministic
6.43.163	Validate fully loaded time < 3s	Fast	Heavy	Deterministic
6.43.164	Detect heavy third‑party scripts	None	Heavy	Deterministic
6.43.165	Validate image formats optimized (WebP/AVIF)	Optimized	Medium	Deterministic
6.43.166	Detect oversized image resources	None	Medium	Deterministic
6.43.167	Validate WebFont usage efficient	Efficient	Medium	Deterministic
6.43.168	Validate caching strategy for static assets	Effective	Medium	Deterministic
6.43.169	Detect heavy synchronous JS	None	Medium	Deterministic
6.43.170	Validate total JS payload < 200kb (compressed)	Clean	Medium	Deterministic

Image/Video/Resource Performance

ID	Check	Passes when	Weight	Runs
6.43.171	Validate hero images correctly compressed	Compressed	Heavy	Deterministic
6.43.172	Validate video lazy-loading implemented	Implemented	Heavy	Deterministic
6.43.173	Detect oversized background images	None	Heavy	Deterministic
6.43.174	Validate correct srcset for responsive images	Correct	Medium	Deterministic
6.43.175	Validate images preloaded only when necessary	Correct	Medium	Deterministic
6.43.176	Detect images missing width/height	None	Medium	Deterministic
6.43.177	Validate video codecs modern (H.265/VP9/AV1 if possible)	Modern	Medium	Deterministic
6.43.178	Validate image aspect ratios correct	Correct	Medium	Deterministic
6.43.179	Detect auto‑playing videos without async controls	None	Medium	Deterministic
6.43.180	Validate LCP media uses efficient delivery (CDN, optimization)	Efficient	Medium	Deterministic

JS/CSS Execution & AI-Discoverability Performance

ID	Check	Passes when	Weight	Runs
6.43.181	Validate JS bundle splitting	Split	Heavy	Deterministic
6.43.182	Validate third‑party scripts loaded async/defer	Async	Heavy	Deterministic
6.43.183	Detect legacy JavaScript inflating parse times	None	Heavy	Deterministic
6.43.184	Validate CSS critical path minimized	Minimized	Heavy	Deterministic
6.43.185	Validate hydration delays not impacting usable content	Minimal	Medium	Deterministic
6.43.186	Validate dynamic widgets (reviews, CTAs) not blocking rendering	Clean	Medium	Deterministic
6.43.187	Detect heavy web-component frameworks (shadow DOM bloat)	None	Medium	Deterministic
6.43.188	Validate inline CSS minimized	Minimal	Medium	Deterministic
6.43.189	Validate AI crawlers can fully load rendered resources	Fully loadable	Medium	Deterministic
6.43.190	Detect dynamic content only accessible after user interaction	None	Medium	Deterministic

Final Technical & Performance Scoring

ID	Check	Passes when	Weight	Runs
6.43.191	Validate no index‑blocking technical errors	None	Critical	Deterministic
6.43.192	Validate crawl budget not wasted	Efficient	Critical	Deterministic
6.43.193	Validate structured data load timing optimal	Optimal	Heavy	Deterministic
6.43.194	Validate overall Lighthouse Performance score competitive	Competitive	Heavy	Deterministic
6.43.195	Validate asset delivery meets AI‑readiness standards	AI‑ready	Medium	Deterministic
6.43.196	Detect performance regressions across pages	None	Medium	Deterministic
6.43.197	Validate first‑party scripts optimized	Optimized	Medium	Deterministic
6.43.198	Validate CDN edge coverage appropriate	Appropriate	Medium	Deterministic
6.43.199	Validate reduced exfiltration and heavy‑analytics impact	Reduced	Medium	Deterministic
6.43.200	Compute Final Technical SEO & Performance Score	Finalized	Critical	Deterministic

← 6.41 Entity SEO & Knowledge Graph Alignment6.44 On-Page SEO & Content Structure →

Run these checks on your site

A Deep Audit scores every check here that applies to your page, then an AI pass of up to 150 checks. Included on Pro and Ultra, or $9 for one audit.

See plans

Machine-readable: catalog totals and the full catalog as JSON (IDs, section, weight, status).