AI Crawler Manager

Control which bots can access your website. Changes update the export below instantly.

What are AI crawlers and why do they matter?

AI crawlers collect your content to train language models or power AI answer engines. Blocking them keeps your content out of their datasets without affecting search visibility.

Search engines

0 allowed · 0 blocked

GooglebotGooglebot
Google Search's crawler — controls your visibility in Google.
Inherited
BingbotBingbot
Microsoft Bing's crawler — controls visibility in Bing.
Inherited
SlurpSlurp
Yahoo Search's crawler.
Inherited

AI training crawlers

0 allowed · 0 blocked

GPTBotGPTBot
Collects content to train OpenAI's models (e.g. ChatGPT).
Inherited
ClaudeBotClaudeBot
Collects content to train Anthropic's Claude models.
Inherited
CCBotCCBot
Builds the Common Crawl public dataset used to train many AI models.
Inherited
BytespiderBytespider
ByteDance's crawler, used for AI training.
Inherited
Meta-ExternalAgentmeta-externalagent
Meta's crawler used for AI training.
Inherited

AI search crawlers

0 allowed · 0 blocked

PerplexityBotPerplexityBot
Fetches pages for Perplexity's AI answer engine.
Inherited
Google-ExtendedGoogle-Extended
Controls Google using your content to train Gemini and other AI models.
Inherited
Applebot-ExtendedApplebot-Extended
Controls Apple using your content for AI training.
Inherited

SEO crawlers

0 allowed · 0 blocked

AhrefsBotAhrefsBot
Ahrefs' SEO backlink crawler.
Inherited
SemrushBotSemrushBot
Semrush's SEO analytics crawler.
Inherited
MJ12botMJ12bot
Majestic's backlink crawler.
Inherited

Social crawlers

0 allowed · 0 blocked

TwitterbotTwitterbot
Generates link previews for X/Twitter.
Inherited
facebookexternalhitfacebookexternalhit
Generates link previews for Facebook.
Inherited

Other / data crawlers

0 allowed · 0 blocked

AmazonbotAmazonbot
Amazon's crawler (used by Alexa and AI products).
Inherited
cohere-aicohere-ai
Cohere's crawler for AI products.
Inherited
DiffbotDiffbot
Diffbot's structured-data crawler.
Inherited
ImagesiftBotImagesiftBot
ImageSift's image crawler.
Inherited

Generated robots.txt

0 allowed · 0 blocked

robots.txt block

# robots.txt generated by Robots.txt Studio

# All other crawlers
User-agent: *
Allow: /

Recommendations

GPTBot has no explicit policy. Set Allow or Block to control AI access deliberately.

ClaudeBot has no explicit policy. Set Allow or Block to control AI access deliberately.

CCBot has no explicit policy. Set Allow or Block to control AI access deliberately.

Bytespider has no explicit policy. Set Allow or Block to control AI access deliberately.

More tools

Robots.txt Generator

Build a valid robots.txt from presets and crawler toggles — no syntax required.

Open

Robots.txt Validator

Catch syntax errors and best-practice issues, with a health score.

Open

Robots.txt Analyzer

Fetch and audit any site's live robots.txt in one report.

Open

Related guides

AI Crawler Directory

Every AI crawler compared.

Open

Block GPTBot

OpenAI's training crawler.

Open

Crawler Directory

All crawlers, search to AI.

Open