Robots.txt Analyzer
Enter a domain and get its live robots.txt audited in one report: which crawlers are blocked, how AI bots are handled, whether a sitemap is declared, and a visibility score with prioritized findings. Works on your own site or any competitor's.
What the analysis shows
- Crawler access — which search engines and AI bots can reach the site.
- AI crawler status — whether GPTBot, ClaudeBot, PerplexityBot and others are blocked.
- Sitemap check — is a sitemap declared, and is it an absolute https URL?
- Risk findings — accidental Disallow: /, blocked assets, conflicts.
- A visibility health score summarizing it all.
Built for messy real-world files
Live robots.txt files are rarely clean. The analyzer handles redirects, missing files, wrong content types, and oversized files gracefully — so you get a useful verdict instead of an error, whatever the server returns.
Great for competitor research
How to analyze a site
- Enter any domain — no need to paste the file.
- The analyzer fetches the live /robots.txt and parses it.
- Review crawler access, AI status, sitemap, and the health score.
- Act on the prioritized findings — fix issues in the Generator, then re-analyze.
How do I analyze a website's robots.txt?
Enter the domain into the analyzer. It fetches the live robots.txt, parses it, and reports which crawlers are blocked, how AI bots are handled, whether a sitemap is declared, and an overall visibility score.
Can I check a competitor's robots.txt?
Yes. The analyzer works on any public domain. Enter a competitor's URL to see whether they block AI crawlers, which sections they disallow, and any issues in their file.
What does the visibility health score mean?
It's a summary metric combining crawler access, AI crawler handling, sitemap presence, and risk findings into a single score, so you can gauge a file's health at a glance before reading the details.
Is the robots.txt analyzer free?
Yes. You can analyze any live domain for free with no sign-up. It handles redirects, missing files, and unusual responses gracefully.
Robots.txt Analyzer
Fetch and audit any site's live robots.txt in one report.
Robots.txt Studio Editorial · Technical SEO & crawling
We build robots.txt tooling and parse thousands of real-world files. Guides are written by practitioners and reviewed against the Google and RFC 9309 specifications.