Generate robots.txt

Choose which crawlers can access your site — we'll generate the rules automatically.

Starter template

Search engines

GooglebotGooglebot
BingbotBingbot
SlurpSlurp

AI training crawlers

GPTBotGPTBot
ClaudeBotClaudeBot
CCBotCCBot
BytespiderBytespider
Meta-ExternalAgentmeta-externalagent

AI search crawlers

PerplexityBotPerplexityBot
Google-ExtendedGoogle-Extended
Applebot-ExtendedApplebot-Extended

SEO crawlers

AhrefsBotAhrefsBot
SemrushBotSemrushBot
MJ12botMJ12bot

Social crawlers

TwitterbotTwitterbot
facebookexternalhitfacebookexternalhit

Other / data crawlers

AmazonbotAmazonbot
cohere-aicohere-ai
DiffbotDiffbot
ImagesiftBotImagesiftBot

Advanced rules

Disallow paths

/wp-admin/

Sitemap URL

Live preview

robots.txt

# robots.txt generated by Robots.txt Studio
# Template: Blog

# AI training crawlers (blocked)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Bytespider
User-agent: meta-externalagent
Disallow: /

# AI search crawlers (blocked)
User-agent: PerplexityBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

# Other / data crawlers (blocked)
User-agent: Amazonbot
User-agent: cohere-ai
User-agent: Diffbot
User-agent: ImagesiftBot
Disallow: /

# All other crawlers
User-agent: *
Allow: /
Disallow: /wp-admin/

Valid · health 96/100

1 warnings · 0 recommendations

What this file does

By default, crawlers are allowed, with some sections blocked. 3 crawlers have their own specific rules. 5 of 5 major AI crawlers are blocked. No sitemap is declared. It scores 96/100 (good): 0 errors, 1 warning and 0 recommendations.

More tools

Robots.txt Validator

Catch syntax errors and best-practice issues, with a health score.

Open

Robots.txt Analyzer

Fetch and audit any site's live robots.txt in one report.

Open

URL Tester

See whether a URL is blocked or allowed for any crawler, with a rule trace.

Open

Related guides

robots.txt Examples