Validate robots.txt
Diagnose issues, check crawler configuration, and get a health score.
Health good
- Syntax
- 100
- AI config
- 60
- Best practice
- 78
Crawl-delay of 0
Line 1The group for * sets Crawl-delay: 0, which has no effect. Remove it or set a positive value.
Sitemap uses HTTP
Line 9Sitemap on line 9 uses http://. Prefer https:// for your sitemap URL.
ClaudeBot inherits the default rules
ClaudeBot (Anthropic) has no explicit group and follows your "User-agent: *" rules. Add an explicit group if you want a different policy for it.
PerplexityBot inherits the default rules
PerplexityBot (Perplexity) has no explicit group and follows your "User-agent: *" rules. Add an explicit group if you want a different policy for it.
CCBot inherits the default rules
CCBot (Common Crawl) has no explicit group and follows your "User-agent: *" rules. Add an explicit group if you want a different policy for it.
Google-Extended inherits the default rules
Google-Extended (Google) has no explicit group and follows your "User-agent: *" rules. Add an explicit group if you want a different policy for it.
No comments or documentation
Adding comments (lines starting with #) explains your intent and helps future maintainers.
Robots.txt Generator
Build a valid robots.txt from presets and crawler toggles — no syntax required.
OpenRobots.txt Analyzer
Fetch and audit any site's live robots.txt in one report.
OpenURL Tester
See whether a URL is blocked or allowed for any crawler, with a rule trace.
Open