Fetch any site's robots.txt, see which search and AI crawlers it allows, then fix it and download the corrected file.
No robots.txt analyzed yet
Enter a domain above to fetch and audit its robots.txt — crawler visibility, AI access, and issues in one report.
Enter a domain and the server fetches https://<domain>/robots.txt, follows up to five redirects, and runs the same parser, validator, and URL tester as the other tools. The search visibility score is 70% search-engine access and 30% file health. AI search and AI training access are listed separately and are not scored, because blocking a training crawler is a choice, not a defect.
| Server response | What Google does | What we report |
|---|---|---|
| 200, plain text | Applies the rules | The file's real rules and a score |
| 200, HTML page | Finds no valid rules | No rules found (allow-all), with a warning |
| 404, 410, 401, 403 | Treats as no robots.txt | No robots.txt: compliant crawlers may crawl everything |
| 429 or 5xx | Pauses crawling, then uses a cached copy | Policy unknown — no score |
| Timeout or DNS failure | Treated like a 5xx | Policy unknown — no score |
Yes. robots.txt is public; enter any domain.
The request timed out or the server answered 429 or 5xx. That doesn't mean everything is allowed: Google pauses crawling while robots.txt returns server errors, so fix that first.
The score measures search-engine access and file health. AI access is shown beside it as your policy, so a deliberate opt-out isn't penalized.
Robots.txt Generator
Build a valid robots.txt from presets and crawler toggles — no syntax required.
OpenRobots.txt Validator
Catch syntax errors and best-practice issues, with a health score.
OpenURL Tester
See whether a URL is blocked or allowed for any crawler, with a rule trace.
Open