robots.txt for SEO: Best Practices
robots.txt is a precision instrument in technical SEO — powerful for crawl control, dangerous when misused for indexing. This guide covers why it matters, when you need one, the practices that help, the traps that hurt, and a checklist to run before every deploy.
Crawling vs indexing — the core distinction
This is the most important concept in robots.txt SEO. robots.txt controls crawling (whether a bot fetches a URL). It does not control indexing (whether a URL appears in results).
Disallow does not equal noindex
| Goal | Use |
|---|---|
| Stop a page appearing in Google | noindex meta tag / X-Robots-Tag (allow crawling) |
| Stop crawlers fetching a section | robots.txt Disallow |
| Keep content out of AI datasets | robots.txt Disallow for AI user-agents |
Why robots.txt matters (and when you need one)
Search engines spend a limited amount of crawling on each site. On large sites, letting crawlers wander through internal search results, faceted filters, and endless calendars means important pages are revisited less often. Small sites rarely hit that limit.
- You need one if you want to keep crawlers out of near-infinite URL spaces (e.g. /search, /*?sort=).
- You need one if you want to set a policy for AI training or AI search crawlers.
- It's the standard place to advertise your sitemap.
- A small site that wants everything crawled can skip it — a missing file means allow-all — though a file with just a Sitemap line is good practice.
Failures are silent
Best practices that help SEO
- Disallow infinite/low-value URL spaces: internal search, faceted filters, session parameters.
- Always declare your sitemap with an absolute https URL.
- Keep crawler-specific groups (e.g. AI bots) explicit so policy is intentional, not inherited.
- Re-audit after migrations and CMS changes — replatforming often regenerates robots.txt.
- Prefer canonical tags and parameter handling for duplicate content; use robots.txt for crawl-budget control.
Anti-patterns that hurt SEO
Blocking CSS and JavaScript
Google renders pages to evaluate them. Disallowing assets produces broken renders and can lower rankings.
Using Disallow to remove pages
It prevents crawling, not indexing — and blocks Google from seeing your noindex. Allow crawling + noindex instead.
Blocking pages that have inbound links
They can still appear as URL-only results. Use noindex.
Leaving Disallow: / in production
The classic staging leak that deindexes a whole site.
AI crawlers and SEO are separate decisions
Blocking GPTBot or opting out via Google-Extended has no effect on Google Search — Google documents that Google-Extended is not a ranking signal, and GPTBot is OpenAI's crawler, not Google's. AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) are a separate choice: blocking them keeps you out of those assistants' cited answers.
See how to block AI crawlers for training-vs-search rules you can copy, and manage it in the AI Crawler Manager.
Pre-deploy checklist
- No accidental Disallow: / in the production file.
- CSS/JS and image directories are crawlable.
- Sitemap line present, absolute, https, and resolves.
- AI crawler policy is intentional (explicit allow or block).
- Validated for syntax, then spot-tested on 2–3 real URLs.
Run the live file through the Analyzer for a health score and findings, and verify key URLs in the URL Tester.
Should I block a page with robots.txt or noindex?
If you want it out of Google, use noindex and let Google crawl the page so it can see the tag. Use robots.txt Disallow only when you want to prevent crawling (e.g. crawl-budget control or AI bots), not to control indexing.
Does Disallow remove a page from Google?
No. It stops crawling, but the URL can still be indexed without content if it's linked elsewhere. To remove it, allow crawling and add noindex, or use a removal request.
Does robots.txt affect crawl budget?
Yes, on large sites. Disallowing low-value URL spaces leaves more crawling for important pages. Most small sites never hit a crawl-budget limit.
Should I disallow CSS and JavaScript?
No. Google needs to render pages with their CSS and JS. Blocking assets leads to incomplete rendering and can harm rankings.
Is robots.txt good or bad for SEO?
It's good when used for crawl control and neutral-to-helpful for rankings. It becomes bad only when misused — for example, blocking assets or using Disallow where noindex is needed.
Robots.txt Analyzer
Fetch and audit any site's live robots.txt in one report.
Robots.txt Studio Editorial · Technical SEO & crawling
We build and maintain the robots.txt parser, validator, and tester behind these tools. Guides are checked against Google Search Central documentation and RFC 9309, and their worked examples are verified by the same engine.