robots.txt Syntax
robots.txt has a small vocabulary: a handful of directives, two wildcard characters, and a grouping rule. This page is a precise reference for each one, plus the file-format rules (encoding, size, content type) and the gotchas that cause real outages.
The directives
| Directive | Purpose |
|---|---|
User-agent | Starts a group; names the crawler the following rules apply to (* = all). |
Disallow | A path prefix the crawler must not request. Empty value = allow everything. |
Allow | A path prefix that overrides a broader Disallow (used for exceptions). |
Sitemap | Absolute URL of an XML sitemap. File-level, not tied to a group. |
Crawl-delay | Seconds between requests. Honored by Bing/Yandex; ignored by Google. |
Groups and User-agent
Rules are organized into groups. A group is one or more User-agent lines followed by Allow/Disallow rules. Consecutive User-agent lines share the same rule block.
User-agent: GPTBot
User-agent: CCBot
Disallow: /Path rules: Allow and Disallow
Paths are matched as prefixes from the start of the URL path. A trailing slash matters: /admin matches /administrator, while /admin/ matches only things inside that directory.
User-agent: *
Disallow: /tmp/
Disallow: /*.json$
Allow: /tmp/public/Empty Disallow means allow-all
Wildcards: * and $
- * matches any sequence of characters, including none. /private*/ matches /private/ and /private-data/.
- $ anchors to the end of the path. /*.pdf$ matches URLs that end in .pdf.
- These are the only two special characters. Everything else is literal.
File format, encoding, and delivery
- One directive per line, written Field: value. Field names are case-insensitive.
- Group order doesn't change behavior — each crawler uses the one group that best matches it, wherever it appears.
- Rule order inside a group doesn't matter either: the longest matching path wins.
- Sitemap lines and # comments can go anywhere; Sitemap is not part of any group.
- Save as UTF-8 plain text. LF or CRLF line endings both work.
- Serve it at /robots.txt with status 200 and Content-Type: text/plain.
- Google reads the first 500 KiB and ignores the rest.
# file-level comment
User-agent: * # default group
Disallow: /admin/
User-agent: GPTBot # crawler-specific group
Disallow: /
Sitemap: https://example.com/sitemap.xmlTwo directives on one line
User-agent: * Disallow: / is one malformed line, not two rules. Give each directive its own line.
Copying from a word processor
Curly quotes and rich-text characters break directives. Edit robots.txt as plain text.
An HTML page at /robots.txt
A 200 HTML page (soft 404 or app shell) contains no valid rules, so crawlers treat the site as unrestricted.
Syntax mistakes that break files
Missing the colon
Disallow /admin (no colon) is not a valid directive and is ignored.
Relative sitemap URL
Sitemap: /sitemap.xml is invalid — it must be a full absolute URL.
Crawl-delay: 0
Has no effect and Google ignores Crawl-delay entirely. Use Search Console crawl settings instead.
Catch all of these automatically with the Validator.
Is robots.txt case sensitive?
User-agent names are matched case-insensitively. Directive names are case-insensitive too. But URL paths in Allow/Disallow are case-sensitive: /Admin and /admin are different paths.
What does Disallow: / mean?
It blocks the entire site for the matching crawler — every path. Use it deliberately (for example, to block an AI crawler).
What does the * wildcard do in robots.txt?
It matches any run of characters within a path, so /*.pdf matches any URL containing .pdf. It was an informal extension for years and is now part of the standard (RFC 9309); Google and Bing support it.
Does the order of rules matter in robots.txt?
No. A crawler picks the most specific user-agent group, then applies the longest matching path rule. Neither depends on line order.
What is the $ symbol in robots.txt?
It anchors a pattern to the end of the URL. /page$ matches /page but not /page/sub, /page2, or /page?ref=1 — the query string counts as part of the URL.
Robots.txt Validator
Catch syntax errors and best-practice issues, with a health score.
Robots.txt Studio Editorial · Technical SEO & crawling
We build and maintain the robots.txt parser, validator, and tester behind these tools. Guides are checked against Google Search Central documentation and RFC 9309, and their worked examples are verified by the same engine.
Comments and Sitemap
Anything after a # is a comment (whole-line or inline). The Sitemap directive takes a full absolute URL and can appear multiple times.
For where the Sitemap line should go and how to list several, see robots.txt sitemap.