Block ClaudeBot in robots.txt
ClaudeBot is Anthropic's web crawler, used to gather training data for the Claude models. If you want to opt out of Claude training, a short robots.txt group stops it — with no impact on your search visibility.
What ClaudeBot does
ClaudeBot crawls public web pages to build training data for Anthropic's Claude models. Like other reputable AI crawlers, it identifies itself with a stable user-agent and follows robots.txt. It is distinct from Claude-SearchBot, which fetches pages so Claude can cite current results when answering.
| Property | Value |
|---|---|
| User-agent | ClaudeBot |
| Operator | Anthropic |
| Purpose | Model training data collection |
| Honors robots.txt | Yes (per Anthropic docs) |
| Affects Google ranking | No |
Why site owners care
The decision mirrors GPTBot: publishers protecting licensed content, businesses guarding proprietary writing, and large sites managing crawl load typically block training crawlers. Because the rule is reversible, many sites block by default and revisit later.
Robots.txt AnalyzerAnalyze a siteHow to block ClaudeBot
Block ClaudeBot across your whole site:
User-agent: ClaudeBot
Disallow: /Block ClaudeBot alongside other AI crawlers by listing several user-agents in one group:
User-agent: ClaudeBot
User-agent: GPTBot
User-agent: CCBot
Disallow: /
User-agent: *
Allow: /Explicitly allow ClaudeBot with an empty Disallow:
User-agent: ClaudeBot
Disallow:A note on older Anthropic agents
Anthropic has previously used other tokens (such as anthropic-ai and Claude-Web). Current guidance consolidates on ClaudeBot for crawling and Claude-SearchBot for search. If you want belt-and-suspenders coverage, you can list the legacy tokens too — they simply won't match anything if unused.
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Claude-Web
Disallow: /Common mistakes
Confusing the two Anthropic bots
ClaudeBot is training; Claude-SearchBot fetches pages for cited answers. Block them independently.
Wrong casing or spelling
User-agent matching is case-insensitive, but the token must still read ClaudeBot — not Claude or ClaudeAI.
Expecting it to remove indexed pages
Blocking ClaudeBot affects training only, never search indexing.
Putting the rule under the wrong group
Disallow: / belongs under User-agent: ClaudeBot, not the catch-all *.
How do I block ClaudeBot?
Add User-agent: ClaudeBot followed by Disallow: / to robots.txt. Anthropic documents that ClaudeBot respects robots.txt, so this opts your site out of Claude training.
Will blocking ClaudeBot affect my Google ranking?
No. ClaudeBot is Anthropic's crawler and is unrelated to Googlebot. Blocking it has no effect on search rankings or indexing.
What is the difference between ClaudeBot and Claude-SearchBot?
ClaudeBot collects training data for Anthropic's models. Claude-SearchBot fetches live pages so Claude can cite them when answering questions. They are separate user-agents and can be controlled separately.
Should I also block anthropic-ai and Claude-Web?
Those are older or legacy tokens. Listing them alongside ClaudeBot does no harm and adds coverage, but ClaudeBot is the current crawler to focus on.
Robots.txt Validator
Catch syntax errors and best-practice issues, with a health score.
Robots.txt Studio Editorial · Technical SEO & crawling
We build robots.txt tooling and parse thousands of real-world files. Guides are written by practitioners and reviewed against the Google and RFC 9309 specifications.