Generate robots.txt

Choose which crawlers can access your site — we'll generate the rules automatically.

Starter template

Search engines

3/3 allowed

GooglebotGooglebot
BingbotBingbot
SlurpSlurp

AI training crawlers

0/5 allowed

GPTBotGPTBot
ClaudeBotClaudeBot
CCBotCCBot
BytespiderBytespider
Meta-ExternalAgentmeta-externalagent

AI search crawlers

0/3 allowed

PerplexityBotPerplexityBot
Google-ExtendedGoogle-Extended
Applebot-ExtendedApplebot-Extended

SEO crawlers

3/3 allowed

AhrefsBotAhrefsBot
SemrushBotSemrushBot
MJ12botMJ12bot

Social crawlers

2/2 allowed

TwitterbotTwitterbot
facebookexternalhitfacebookexternalhit

Other / data crawlers

0/4 allowed

AmazonbotAmazonbot
cohere-aicohere-ai
DiffbotDiffbot
ImagesiftBotImagesiftBot

Advanced rules

Disallow paths

/wp-admin/

Sitemap URL

Live preview

robots.txt

# robots.txt generated by Robots.txt Studio
# Template: Blog

# AI training crawlers (blocked)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Bytespider
User-agent: meta-externalagent
Disallow: /

# AI search crawlers (blocked)
User-agent: PerplexityBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

# Other / data crawlers (blocked)
User-agent: Amazonbot
User-agent: cohere-ai
User-agent: Diffbot
User-agent: ImagesiftBot
Disallow: /

# All other crawlers
User-agent: *
Allow: /
Disallow: /wp-admin/

Valid · health 95/100

1 warnings · 1 recommendations

What this file does

By default, crawlers are allowed, with some sections blocked. 3 crawlers have their own specific rules. 5 of 5 major AI crawlers are blocked. No sitemap is declared. It scores 95/100 (good): 0 errors, 1 warning and 1 recommendation.

Next steps