Robots.txt Generator
Writing robots.txt by hand is easy to get subtly wrong. This generator builds a valid file from presets and simple toggles — search engines, AI crawlers, blocked paths, and your sitemap — and validates the output as you go. No sign-up, no syntax knowledge required.
Why use a generator
- Avoid the costly typos — a misplaced Disallow: / can deindex a whole site.
- Get AI crawler rules right (GPTBot, ClaudeBot, Google-Extended) without memorizing user-agent tokens.
- See live validation and a plain-English explanation as you build.
- Start from a preset for blogs, SaaS, e-commerce, publishers, or docs.
How it works
- Pick a preset that matches your site, or start blank.
- Toggle which search engines and AI crawlers may access your site.
- Add any paths to disallow and your sitemap URL.
- Copy the generated, validated robots.txt and deploy it to your site root.
Example output
A typical result — allow search engines, block AI training crawlers, declare the sitemap:
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
Disallow: /
User-agent: *
Allow: /
Disallow: /admin/
Sitemap: https://example.com/sitemap.xmlCommon use cases
New site launch
Generate a safe default that allows search engines and points to your sitemap.
Opting out of AI
Block the major AI training crawlers in one step while staying fully indexable.
E-commerce crawl control
Disallow cart, checkout, and faceted URLs to protect crawl budget.
Fixing a broken file
Rebuild cleanly instead of patching a file with conflicting rules.
Validate before you ship
Is this robots.txt generator free?
Yes. The generator is free and requires no sign-up. Build your file, copy it, and deploy it to your site root at /robots.txt.
Does the generator block AI crawlers?
Yes. You can toggle AI crawlers like GPTBot, ClaudeBot, CCBot, and Google-Extended on or off, and the generator writes the correct user-agent groups for you.
Will the generated robots.txt be valid?
Yes. Output is validated as you build, so you get a syntactically correct file that follows robots.txt best practices, including an absolute sitemap URL.
Where do I put the generated robots.txt?
Place it at the root of your domain so it's served at https://yourdomain.com/robots.txt. It must be at the root — crawlers don't read robots.txt from subdirectories.
Robots.txt Generator
Build a valid robots.txt from presets and crawler toggles — no syntax required.
Robots.txt Studio Editorial · Technical SEO & crawling
We build robots.txt tooling and parse thousands of real-world files. Guides are written by practitioners and reviewed against the Google and RFC 9309 specifications.