robots.txt Sitemap
The Sitemap directive tells crawlers where your XML sitemap lives. It's the easiest win in a robots.txt file — and one of the most common to get slightly wrong with a relative URL.
robots.txt vs sitemap: restriction vs suggestion
robots.txt tells crawlers which paths they may not request. An XML sitemap lists the URLs you want crawled. One subtracts, the other adds — most sites benefit from both, and the Sitemap line in robots.txt is where they connect.
| robots.txt | XML sitemap | |
|---|---|---|
| Purpose | Restrict crawling | Advertise URLs to crawl |
| Format | Plain-text directives | XML list of URLs |
| Location | /robots.txt (fixed) | Any URL, e.g. /sitemap.xml |
| Guarantees indexing? | No | No — it's a hint |
Don't contradict yourself
The Sitemap directive
Sitemap is a file-level directive: it isn't tied to any User-agent group and can appear anywhere in the file. It must be a full, absolute URL.
Sitemap: https://example.com/sitemap.xmlRelative URLs are invalid
Multiple sitemaps
You can list as many sitemaps as you like — one per line. Large sites often split sitemaps by section or content type.
Sitemap: https://example.com/sitemap-pages.xml
Sitemap: https://example.com/sitemap-products.xml
Sitemap: https://example.com/sitemap-blog.xmlSitemap index files
If you have many sitemaps, reference a single sitemap index instead — it links to the individual sitemaps, keeping robots.txt clean.
Sitemap: https://example.com/sitemap_index.xmlWhere to put it
Anywhere works, but most teams put Sitemap lines at the very top or very bottom of the file for readability. Order has no effect on behavior.
User-agent: *
Allow: /
Disallow: /admin/
Sitemap: https://example.com/sitemap.xmlThe Generator adds and positions the Sitemap line for you, and the Validator warns if it's missing or uses http instead of https.
Common sitemap mistakes
Relative path
Use the absolute URL, not /sitemap.xml.
http on an https site
Match your canonical protocol — declare the https:// sitemap URL.
Sitemap as the only submission method
robots.txt discovery is passive; also submit your sitemap in Google Search Console for reporting.
Pointing to a 404
Make sure the sitemap URL actually resolves before referencing it.
Can I list multiple sitemaps in robots.txt?
Yes. Add one Sitemap line per sitemap. For many sitemaps, reference a single sitemap index file that links to the rest.
Does the Sitemap line need to be a full URL?
Yes. It must be an absolute URL including protocol and host (https://example.com/sitemap.xml). Relative paths are ignored.
Where should the Sitemap line go in robots.txt?
Anywhere — it's file-level and not part of any group. Top or bottom of the file is conventional for readability.
Is robots.txt the only way to submit a sitemap?
No. The Sitemap directive helps crawlers discover it passively, but you should also submit your sitemap directly in Google Search Console and Bing Webmaster Tools.
Robots.txt Generator
Build a valid robots.txt from presets and crawler toggles — no syntax required.
Robots.txt Studio Editorial · Technical SEO & crawling
We build and maintain the robots.txt parser, validator, and tester behind these tools. Guides are checked against Google Search Central documentation and RFC 9309, and their worked examples are verified by the same engine.