AI Crawler Manager

Control which bots can access your website. Changes update the export below instantly.

What are AI crawlers and why do they matter?

AI crawlers collect your content to train language models or power AI answer engines. Blocking them keeps your content out of their datasets without affecting search visibility.

Start from a template

Pick a template to start, then fine-tune any crawler below — your choices update the file live.

Search engines

0 allowed · 0 blocked

GooglebotGooglebot
Google Search's crawler — controls your visibility in Google.
Inherited
BingbotBingbot
Microsoft Bing's crawler — controls visibility in Bing.
Inherited
SlurpSlurp
Yahoo Search's crawler.
Inherited

AI training crawlers

0 allowed · 0 blocked

GPTBotGPTBot
Collects content to train OpenAI's models (e.g. ChatGPT).
Inherited
ClaudeBotClaudeBot
Collects content to train Anthropic's Claude models.
Inherited
CCBotCCBot
Builds the Common Crawl public dataset used to train many AI models.
Inherited
BytespiderBytespider
ByteDance's crawler, used for AI training.
Inherited
Meta-ExternalAgentmeta-externalagent
Meta's crawler used for AI training.
Inherited

AI search crawlers

0 allowed · 0 blocked

OAI-SearchBotOAI-SearchBot
Surfaces and links sites in ChatGPT search answers. Separate from GPTBot; not used for training.
Inherited
Claude-SearchBotClaude-SearchBot
Indexes pages to improve Claude's web search results. Separate from ClaudeBot (training).
Inherited
PerplexityBotPerplexityBot
Surfaces and links sites in Perplexity answers. Perplexity says it is not used to train foundation models.
Inherited

AI usage controls (no crawler)

0 allowed · 0 blocked

Google-ExtendedGoogle-Extended
Control token (no separate crawler): governs Gemini training and grounding use of content Googlebot fetched. No effect on Google Search.
Inherited
Applebot-ExtendedApplebot-Extended
Control token (no separate crawler): governs Apple's use of Applebot-fetched content to train its foundation models. No effect on Apple search.
Inherited

SEO crawlers

0 allowed · 0 blocked

AhrefsBotAhrefsBot
Ahrefs' SEO backlink crawler.
Inherited
SemrushBotSemrushBot
Semrush's SEO analytics crawler.
Inherited
MJ12botMJ12bot
Majestic's backlink crawler.
Inherited

Social crawlers

0 allowed · 0 blocked

TwitterbotTwitterbot
Generates link previews for X/Twitter.
Inherited
facebookexternalhitfacebookexternalhit
Generates link previews for Facebook.
Inherited

Other / data crawlers

0 allowed · 0 blocked

AmazonbotAmazonbot
Amazon's crawler (used by Alexa and AI products).
Inherited
cohere-aicohere-ai
Cohere's crawler for AI products.
Inherited
DiffbotDiffbot
Diffbot's structured-data crawler.
Inherited
ImagesiftBotImagesiftBot
ImageSift's image crawler.
Inherited

Generated robots.txt

Live

robots.txt block

# robots.txt generated by Robots.txt Studio

# All other crawlers
User-agent: *
Allow: /

0 allowed · 0 blocked

Recommendations

GPTBot has no explicit policy. Set Allow or Block to control AI access deliberately.

ClaudeBot has no explicit policy. Set Allow or Block to control AI access deliberately.

CCBot has no explicit policy. Set Allow or Block to control AI access deliberately.

Bytespider has no explicit policy. Set Allow or Block to control AI access deliberately.

Next steps

Open the full Generator

Add disallow paths, a sitemap, and search-engine rules.

Open

More tools

Robots.txt Generator

Build a valid robots.txt from presets and crawler toggles — no syntax required.

Open

Robots.txt Validator

Catch syntax errors and best-practice issues, with a health score.

Open

Robots.txt Analyzer

Fetch and audit any site's live robots.txt in one report.

Open

Related guides

AI Crawler Directory

Every AI crawler compared.

Open

How to Block AI Crawlers

The complete blocking playbook.

Open

Block GPTBot

OpenAI's training crawler.

Open