Compare robots.txt
Diff two sites (or two versions): score changes, findings fixed or introduced, and per-crawler visibility — including a dedicated AI-crawler report.
Site A (before)
Site B (after)
Score changes (A → B)
Overall health
9498+4
Crawler visibility
9691-5
Syntax
1001000
SEO
1001000
AI Crawlers
7590+15
Maintainability
9497+3
Security
1001000
AI crawler visibility (A → B)
- GPTBotAllowedBlocked
- ClaudeBotAllowedAllowed
- CCBotAllowedBlocked
- BytespiderAllowedAllowed
- Meta-ExternalAgentAllowedAllowed
- PerplexityBotAllowedAllowed
- Google-ExtendedAllowedBlocked
- Applebot-ExtendedAllowedAllowed
Findings resolved
- No crawler-specific groups
Findings introduced
No new findings.
File diff
Before / After−0+9
| User-agent: * |
| Allow: / |
| Added: User-agent: GPTBot |
| Added: Disallow: / |
| Added: |
| Added: User-agent: CCBot |
| Added: Disallow: / |
| Added: |
| Added: User-agent: Google-Extended |
| Added: Disallow: / |
| Added: |
| Sitemap: https://example.com/sitemap.xml |