robots.txt Templates
Pick a template that matches your site, copy it, and drop it at /robots.txt. Every template here is valid and minimal — open the Generator to tweak paths, crawlers, and your sitemap, or the Validator to check a file you already have.
How to use these templates
- Pick the template that best matches your site type below.
- Copy it and save it as a plain-text file named robots.txt.
- Replace example.com in the Sitemap line with your real domain.
- Upload it to the root of your site so it's served at /robots.txt.
- Validate it, then confirm specific URLs with the URL Tester.
Edit the sitemap line
Allow everything
The safe default for most sites — let every crawler access everything, and point them to your sitemap.
User-agent: *
Disallow:
Sitemap: https://example.com/sitemap.xmlBlog / personal site
Open to search engines, with admin and search-result pages kept out of the crawl.
User-agent: *
Allow: /
Disallow: /search
Disallow: /*?s=
Sitemap: https://example.com/sitemap.xmlWordPress
Recommended WordPress baseline: block wp-admin (except admin-ajax) and keep the rest crawlable.
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/sitemap.xmlE-commerce
Keep cart, checkout, and account pages out of the index while products and categories stay crawlable.
User-agent: *
Allow: /
Disallow: /cart
Disallow: /checkout
Disallow: /account
Disallow: /*?sort=
Disallow: /*?filter=
Sitemap: https://example.com/sitemap.xmlSaaS / web app
Index the marketing site, but keep the authenticated app, API, and internal routes out of search.
User-agent: *
Allow: /
Disallow: /app/
Disallow: /api/
Disallow: /dashboard/
Disallow: /settings/
Sitemap: https://example.com/sitemap.xmlDocumentation site
Fully open to crawlers — docs benefit from maximum discoverability — with the sitemap declared.
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlBlock AI training, keep search
Stay fully visible in Google and Bing while opting out of the major AI training crawlers.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Bytespider
Disallow: /
User-agent: *
Allow: /
Sitemap: https://example.com/sitemap.xmlBlock everything (staging)
For staging or maintenance only — blocks all crawlers from the whole site. Never ship this to production.
User-agent: *
Disallow: /Where do I put the robots.txt file?
robots.txt must live at the root of your domain so it's served at https://yourdomain.com/robots.txt. It only applies to the exact host and protocol it's served from — subdomains need their own file.
Which robots.txt template should I use?
Start from the one matching your platform — WordPress, e-commerce, SaaS, blog, or docs. If your only goal is to keep AI companies from training on your content while staying in Google, use the 'Block AI training, keep search' template.
Will these templates hurt my SEO?
No — they follow best practice. The only template that blocks search engines is 'Block everything (staging)', which is meant for non-production sites. Never deploy that one to a live site you want indexed.
Can I combine templates?
Yes. The Generator lets you merge rules — for example the WordPress baseline plus the AI-blocking group — and keeps the output valid. Paste a template in and adjust from there.
Robots.txt Generator
Build a valid robots.txt from presets and crawler toggles — no syntax required.
Robots.txt Studio Editorial · Technical SEO & crawling
We build robots.txt tooling and parse thousands of real-world files. Guides are written by practitioners and reviewed against the Google and RFC 9309 specifications.