Home / Check catalog / 5. Technical SEO & Index Control / 5.1 robots.txt, Meta Robots & Indexability Signals Suite

5.1 robots.txt, Meta Robots & Indexability Signals Suite

45 checks in Technical SEO & Index Control. Automated Its subsection has a live automated checker in the engine.

IDCheckPasses whenWeightRuns
5.1.01Fetch robots.txtAccessibleCriticalDeterministic
5.1.02Validate robots.txt syntax correctnessValidCriticalDeterministic
5.1.03Validate no accidental Disallow: /NoneCriticalDeterministic
5.1.04Validate correct User-Agent targeting (Googlebot, Bingbot, Adsbot, etc.)CorrectHeavyDeterministic
5.1.05Validate sitemap directives includedPresentMediumDeterministic
5.1.06Detect conflicting allow/disallow patternsNoneMediumDeterministic
5.1.07Validate robots.txt does not block JSAllowedMediumDeterministic
5.1.08Validate robots.txt does not block CSSAllowedMediumDeterministic
5.1.09Validate robots.txt does not block key assets (images/fonts)AllowedMediumDeterministic
5.1.10Validate correct use of Crawl-delay (avoid harming crawl rates)CorrectHeavyDeterministic
5.1.11Extract meta robots from DOMPresent or intentionally absentHeavyDeterministic
5.1.12Detect accidental noindexNoneCriticalDeterministic
5.1.13Detect accidental nofollowNoneHeavyDeterministic
5.1.14Detect accidental noarchiveNoneHeavyDeterministic
5.1.15Detect accidental nosnippetNoneHeavyDeterministic
5.1.16Validate multiple meta robots tags consistencyConsistentMediumDeterministic
5.1.17Extract X-Robots-Tag HTTP headerPresent if neededCriticalDeterministic
5.1.18Detect accidental noindex via HTTP headerNoneCriticalDeterministic
5.1.19Detect conflicting meta robots vs X-Robots-TagNo conflictHeavyDeterministic
5.1.20Validate canonical tag existsPresentHeavyDeterministic
5.1.21Validate canonical tag is absolute URLCorrectHeavyDeterministic
5.1.22Validate canonical is not self-contradictingCorrectHeavyDeterministic
5.1.23Detect canonical loopsNoneMediumDeterministic
5.1.24Detect multiple canonicalsOnly oneMediumDeterministic
5.1.25Validate URL normalization (no mixed casing)CleanMediumDeterministic
5.1.26Validate URL normalization (no trailing slash conflicts)ConsistentMediumDeterministic
5.1.27Validate URL normalization (http→https enforced)RedirectedMediumDeterministic
5.1.28Validate canonicalization across paginationCorrectHeavyDeterministic
5.1.29Validate indexability of paginated pagesCorrectHeavyDeterministic
5.1.30Validate indexability of filtered/parameter pagesControlledHeavyDeterministic
5.1.31Validate AMP pages indexability (if present)CorrectMediumDeterministic
5.1.32Validate alternate/hreflang not blockedAccessibleMediumDeterministic
5.1.33Validate meta viewport not blockedAccessibleMediumDeterministic
5.1.34Validate OG/Twitter metadata not blocked via robotsAccessibleMediumDeterministic
5.1.35Validate structured data pages indexableIndexableMediumDeterministic
5.1.36Validate image/video content not incorrectly disallowedAccessibleMediumDeterministic
5.1.37Validate correct use of max-snippet, max-image-preview, max-video-previewOptimizedHeavyDeterministic
5.1.38Detect pages blocked by server authenticationMinimalMediumDeterministic
5.1.39Detect soft 404s harming indexabilityNoneMediumDeterministic
5.1.40Validate indexability of primary category pagesYesHeavyDeterministic
5.1.41Validate indexability of service/location pagesYesHeavyDeterministic
5.1.42Validate indexability of homepageYesHeavyDeterministic
5.1.43Detect accidental canonicalization to wrong domainNoneMediumDeterministic
5.1.44Validate that search engines can fully render JS contentYesHeavyDeterministic
5.1.45Compute indexability composite score>= 90%CriticalDeterministic

5.2 Crawlability, Internal Graph Reachability & Link Discovery Suite →

Run these checks on your site

A Deep Audit scores every check here that applies to your page, then an AI pass of up to 150 checks. Included on Pro and Ultra, or $9 for one audit.

See plans

Machine-readable: catalog totals and the full catalog as JSON (IDs, section, weight, status).