Claude-SearchBot in robots.txt
Claude-SearchBot is Anthropic's search crawler. It fetches pages so Claude can answer with current web results and cite you as the source. It is not the training crawler — that's ClaudeBot, and the two are decided separately in robots.txt.
What Claude-SearchBot does
When Claude searches the web for a user, Claude-SearchBot is the crawler that retrieves the pages Claude reads and cites. Pages it can access are eligible to be quoted — with attribution — in Claude's answers. Anthropic documents it as a search crawler, distinct from ClaudeBot, which collects model-training data under its own user-agent.
| Property | Value |
|---|---|
| User-agent | Claude-SearchBot |
| Operator | Anthropic |
| Purpose | Fetch and cite pages for Claude's web search |
| Trains AI models | No (that's ClaudeBot) |
| Honors robots.txt | Yes (per Anthropic docs) |
| Affects Google ranking | No |
Source: Anthropic's crawler documentation, which lists each Anthropic user-agent and how to control it.
Should you allow or block it?
Like other AI-search crawlers, this is a visibility trade-off rather than a content-protection question — its output cites and links you.
- Allow it if you want Claude to cite your pages in answers — citations include your link.
- Block it if you want no presence in AI-generated answers, even attributed ones.
- Blocking ClaudeBot does NOT block Claude-SearchBot — separate user-agents, separate groups.
Example robots.txt rules
User-agent: Claude-SearchBot
Disallow:User-agent: Claude-SearchBot
Disallow: /# Opt out of training, stay citable in Claude's web answers
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Disallow:Verify your rule works
Confirm the rule does what you intended: run your domain through the Analyzer to see Claude-SearchBot's resolved access, or paste the file into the URL Tester and test a path against the Claude-SearchBot user-agent.
Robots.txt AnalyzerAnalyze a siteCommon mistakes
Assuming ClaudeBot rules cover it
A Disallow under User-agent: ClaudeBot has no effect on Claude-SearchBot. Each Anthropic crawler needs its own group.
Blocking it to keep content out of training
Claude-SearchBot doesn't collect training data. Blocking it removes you from Claude's cited answers without changing what trains on your content.
Typos in the token
It must be exactly Claude-SearchBot. ClaudeSearchBot or Claude-Search won't match.
What is Claude-SearchBot?
Claude-SearchBot is Anthropic's search crawler. It fetches web pages so Claude can answer questions with current information and cite the source. It is separate from ClaudeBot and is not used for model training.
Does Claude-SearchBot train Claude?
No. Anthropic documents that Claude-SearchBot only retrieves pages for Claude's web search answers. Training data collection happens under the ClaudeBot user-agent, which you control separately in robots.txt.
Should I block Claude-SearchBot?
Only if you don't want your pages cited in Claude's answers. Allowing it keeps you eligible for attributed citations with links, and it has no effect on AI training — many sites allow it while blocking ClaudeBot.
Does Claude-SearchBot respect robots.txt?
Yes. Anthropic states that its crawlers, including Claude-SearchBot, honor robots.txt rules targeting their user-agent token.
Robots.txt Validator
Catch syntax errors and best-practice issues, with a health score.
Robots.txt Studio Editorial · Technical SEO & crawling
We build robots.txt tooling and parse thousands of real-world files. Guides are written by practitioners and reviewed against the Google and RFC 9309 specifications.