Robots.txt Tester

What This Tool Does

Your robots.txt file tells search engine crawlers which parts of your site they may or may not request. A single wrong Disallow line can deindex an entire section of your site. This tool fetches the live robots.txt, parses every user-agent group, and lets you test whether a specific path is allowed for a specific crawler using Google's longest-match precedence rule.

Common Mistakes It Catches

• Accidentally blocking everything — a stray Disallow: / under User-agent: *
• Missing Sitemap directive — crawlers have to guess your URL inventory
• Blocking CSS/JS — breaks how Google renders and ranks the page

Want the Full Picture?

The SEO Audit API checks crawlability plus 81 other signals and returns a scored report. Free tier available.