Robots.txt Validator
Audit, jump to the exact line, fix in one click, and watch the score recover.
Your robots.txt is technically valid, but AI-crawler coverage is the biggest gap — decide allow/block per crawler.
How the robots.txt validator checks your file
Paste a file or fetch it from your domain. The validator parses it the way RFC 9309 and Google describe, then reports each problem with its line, its severity, and a fix you can apply and re-check in place. It runs in your browser — the file you paste is not uploaded.
Checking, validating, and testing are different questions
- Validate: is the file well-formed, and does it contain risky patterns? (this page)
- Test: is one specific URL allowed or blocked for one crawler? Use the URL Tester — a valid file can still block the wrong page.
- Analyze: fetch a live site's file and see which search and AI crawlers it lets in.
What each severity means
| Severity | Meaning | Example |
|---|---|---|
| Error | Crawlers will ignore or misread the line | Disallow /admin/ (no colon) |
| Warning | Valid, but likely not what you intended | User-agent: * + Disallow: / |
| Info | A decision worth making explicitly | No rule for GPTBot (inherits *) |
What it catches: examples
A missing colon silently drops the rule
User-agent: *
Disallow /admin/Write Disallow: /admin/. Without the colon, crawlers skip the line and /admin/ stays crawlable.
The whole site is blocked for every crawler
User-agent: *
Disallow: /Usually a staging file shipped to production. Remove the line, or block only specific paths.
Noindex in robots.txt does nothing
User-agent: *
Noindex: /private/Google stopped supporting it in 2019. Use a noindex meta tag or X-Robots-Tag header, and leave the page crawlable so the tag can be seen.
A relative sitemap URL is ignored
User-agent: *
Allow: /
Sitemap: /sitemap.xmlUse the absolute URL: Sitemap: https://example.com/sitemap.xml.
Blocking CSS or JavaScript hurts rendering
User-agent: *
Disallow: /*.css$Google renders pages like a browser; let it fetch the CSS and JS the page needs.
How do I check if my robots.txt is valid?
Paste the file above or fetch it from your domain. The validator lists every error and warning with its line number and a suggested fix. Apply fixes in place, re-check, then download the corrected file.
What's the difference between a robots.txt checker and a validator?
People use the words interchangeably. Validation checks the file itself (syntax, conflicts, risky rules). To check whether a specific page is reachable, test that URL against a crawler in the URL Tester.
Does a valid robots.txt mean Google will index my pages?
No. robots.txt only controls crawling by crawlers that choose to follow it. A valid file can still block important URLs, and an allowed URL is not guaranteed to be crawled or indexed.
Is the validator free?
Yes. It needs no sign-up, and a pasted file is processed in your browser.
Next steps
- URL TesterConfirm specific URLs are allowed for Googlebot or an AI bot.
- robots.txt syntaxEvery directive, wildcard, and formatting rule.
- robots.txt vs noindexWhy blocking a page doesn't remove it from Google.
More tools & related guides
Robots.txt Generator
Build a valid robots.txt from presets and crawler toggles — no syntax required.
OpenRobots.txt Analyzer
Fetch and audit any site's live robots.txt in one report.
OpenURL Tester
See whether a URL is blocked or allowed for any crawler, with a rule trace.
Open