robots.txt Checker
A robots.txt checker answers two questions: is my file valid, and does it actually block what I think it blocks? This guide covers both — syntax checking and URL testing — plus how to audit a live site and act on what you find.
Two kinds of checking
“Checking” robots.txt means two different things, and you usually want both. Validating confirms the file is well-formed and follows best practices. Testing confirms a specific URL is allowed or blocked for a given crawler.
- Validate → catches syntax errors, conflicts, missing sitemaps, and risky rules.
- Test → answers “is /admin/login blocked for GPTBot?” with the matched rule.
- Analyze → fetches a live site's file and combines validation, AI-crawler status, and a health score.
Validate the syntax
Start by checking the file is valid. A good validator flags errors (missing colons, conflicting rules, malformed sitemaps) and warnings (duplicate rules, Crawl-delay: 0, missing sitemap), and scores overall health.
Paste your file into the Validator to see issues by severity, each with the line number and a fix.
Test specific URLs
Validation tells you the file is well-formed; it doesn't tell you whether your important pages are reachable. For that, test real URLs against real crawlers.
robots.txt:
User-agent: *
Disallow: /admin/
URL: /admin/login
Crawler: Googlebot
→ BLOCKED by Disallow: /admin/The URL Tester shows the verdict, the matched rule, and a step-by-step trace so the result is never a black box.
Audit a live site
To check a site that's already deployed (yours or a competitor's), fetch the live file and audit it in one step — including whether AI crawlers are blocked and whether a sitemap is declared.
Enter any domain in the Analyzer. It handles redirects, missing files, and wrong content types gracefully, then returns a visibility score and prioritized findings.
Acting on what a checker finds
Error: conflicting rules
A path is both allowed and disallowed — decide which you want and remove the other.
Warning: missing sitemap
Add a Sitemap line with the absolute https URL.
Info: AI crawler inherits default
Set an explicit policy for major AI bots so it's intentional.
Once you know what to change, rebuild the file safely in the Generator and re-validate before deploying.
How do I check if my robots.txt is valid?
Paste it into a validator. It parses the file, flags syntax errors and best-practice issues by severity with line numbers, and gives a health score.
How do I test if a URL is blocked by robots.txt?
Use a URL tester: provide the robots.txt, the URL, and a crawler. It applies the real matching rules and returns allowed or blocked along with the rule that decided it.
How do I check a competitor's robots.txt?
Enter their domain into the Analyzer. It fetches the live file and breaks down which crawlers are blocked, how AI bots are handled, and any issues.
What's the difference between a robots.txt checker and a validator?
A validator focuses on whether the file is well-formed and follows best practices. A checker is broader — it usually also tests specific URLs and may fetch and audit a live site. Robots.txt Studio does all three.
Robots.txt Validator
Catch syntax errors and best-practice issues, with a health score.
Robots.txt Studio Editorial · Technical SEO & crawling
We build robots.txt tooling and parse thousands of real-world files. Guides are written by practitioners and reviewed against the Google and RFC 9309 specifications.