Explain robots.txt
Translate your robots.txt into plain English — no technical knowledge required.
all crawlers are allowed, but blocked from: /admin/.
- Blocked from /admin/.
OpenAI's GPTBot is blocked from your entire site.
- Blocked from your entire site (/).
It trains OpenAI's models (such as ChatGPT) on your content.
OpenAI's GPTBot is blocked from your entire site.
It trains Anthropic's Claude models on your content.
Anthropic's ClaudeBot has no rule of its own, so it follows your default settings — it is currently partially blocked.
Add an explicit Allow or Disallow for ClaudeBot to control its access deliberately.
It fetches your pages for Perplexity's AI answer engine.
Perplexity's PerplexityBot has no rule of its own, so it follows your default settings — it is currently partially blocked.
Add an explicit Allow or Disallow for PerplexityBot to control its access deliberately.
It collects data for Common Crawl, a public dataset used to train many AI models.
Common Crawl's CCBot has no rule of its own, so it follows your default settings — it is currently partially blocked.
Add an explicit Allow or Disallow for CCBot to control its access deliberately.
It controls whether Google can use your content to train Gemini and other AI models.
Google-Extended (Google's AI training) has no rule of its own, so it follows your default settings — it is currently partially blocked.
Add an explicit Allow or Disallow for Google-Extended to control its access deliberately.
- A sitemap is declared at https://example.com/sitemap.xml, which helps search engines find your pages.
Robots.txt Generator
Build a valid robots.txt from presets and crawler toggles — no syntax required.
OpenRobots.txt Validator
Catch syntax errors and best-practice issues, with a health score.
OpenRobots.txt Analyzer
Fetch and audit any site's live robots.txt in one report.
Open