Robots.txt Generator

Choose which crawlers can access your site — we'll generate the rules automatically.

Starter template

Search engines

3/3 allowed

GooglebotGooglebot
BingbotBingbot
SlurpSlurp

AI training crawlers

0/5 allowed

GPTBotGPTBot
ClaudeBotClaudeBot
CCBotCCBot
BytespiderBytespider
Meta-ExternalAgentmeta-externalagent

AI search crawlers

0/3 allowed

OAI-SearchBotOAI-SearchBot
Claude-SearchBotClaude-SearchBot
PerplexityBotPerplexityBot

AI usage controls (no crawler)

0/2 allowed

Google-ExtendedGoogle-Extended
Applebot-ExtendedApplebot-Extended

SEO crawlers

3/3 allowed

AhrefsBotAhrefsBot
SemrushBotSemrushBot
MJ12botMJ12bot

Social crawlers

2/2 allowed

TwitterbotTwitterbot
facebookexternalhitfacebookexternalhit

Other / data crawlers

0/4 allowed

AmazonbotAmazonbot
cohere-aicohere-ai
DiffbotDiffbot
ImagesiftBotImagesiftBot

Advanced rules

Disallow paths

/wp-admin/

Sitemap URL

Live preview

robots.txt

# robots.txt generated by Robots.txt Studio
# Template: Blog

# AI training crawlers (blocked)
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Bytespider
User-agent: meta-externalagent
Disallow: /

# AI search crawlers (blocked)
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Disallow: /

# AI usage controls (no crawler) (blocked)
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

# Other / data crawlers (blocked)
User-agent: Amazonbot
User-agent: cohere-ai
User-agent: Diffbot
User-agent: ImagesiftBot
Disallow: /

# All other crawlers
User-agent: *
Allow: /
Disallow: /wp-admin/

Valid · health 95/100

1 warnings · 1 recommendations

What this file does

By default, crawlers are allowed, with some sections blocked. 4 crawlers have their own specific rules. 5 of 5 major AI crawlers are blocked. No sitemap is declared. It scores 95/100 (good): 0 errors, 1 warning and 1 recommendation.

Next steps

Building a robots.txt that does what you intend

Start from a template, decide each crawler group, add the paths you don't want crawled, and copy or download the result. The preview updates as you change settings, so you can see every line before you ship.

Search, AI search, and AI training are separate decisions

Group	Examples	What blocking does
Search engines	Googlebot, Bingbot	Removes you from that search engine over time — rarely what you want
AI search	OAI-SearchBot, Claude-SearchBot, PerplexityBot	Stops those assistants surfacing and linking your pages
AI training	GPTBot, ClaudeBot, CCBot	Asks those operators not to collect your content for model training
AI usage controls	Google-Extended, Applebot-Extended	Opts out of AI training/grounding use; no crawler is blocked and search is unaffected

Before you deploy

Run the file through the Validator and fix any errors.
Test your most important URLs for Googlebot in the URL Tester.
Upload it as /robots.txt at the root of each host (https://example.com/robots.txt). Subdomains need their own file.
After deploying, analyze the live domain to confirm it serves 200 with text/plain.

Frequently asked questions

Is the generator free?

Yes, with no sign-up. Copy the output or download robots.txt.

Does blocking AI crawlers affect Google rankings?

Blocking AI training crawlers or Google-Extended does not affect Google Search, according to Google's documentation. Blocking Googlebot does.

Does robots.txt stop all AI crawlers?

It is a request to compliant crawlers, not access control. Operators that publish their crawlers say they honor it; others may not. Use server-side controls for content that must stay private.

Where do I put the generated file?

At the root of the host as /robots.txt. On WordPress, Shopify, or Next.js the method differs — see the platform guides.