Robots.txt Tester

Check whether a URL is blocked or allowed for any crawler, and see exactly which rule decided it.

robots.txt

URL to test

User-agent

blocked

"GPTBot" is blocked from /admin/login by "Disallow: /".

Matched rule

User-agent: GPTBot

Disallow: /

Rule trace

1. Match user-agent
Matched the most specific group, "User-agent: GPTBot", for "GPTBot".
2. Evaluate path rules
1 rule(s) match "/admin/login": Disallow: /.
3. Apply precedence
Longest match wins; Allow beats Disallow on ties. Winner: "Disallow: /" (line 6).
4. Decision
Blocked by "Disallow: /".

Next steps

How the URL tester decides allowed or blocked

The tester applies the same three steps a compliant crawler does: pick the one group that matches the crawler, find every Allow/Disallow rule that matches the URL path, then apply precedence. The rule trace above shows each step for your input.

The rules that decide every result

One group per crawler: the most specific User-agent match wins. A named group does not inherit rules from User-agent: *.
Longest match wins: the rule with the longest matching path takes precedence, regardless of line order.
Ties go to Allow: when an Allow and a Disallow match with equal length, the URL is allowed.
Paths are case-sensitive and include the query string; $ anchors the end of the URL.
No matching rule means allowed.

Worked examples

Each result below is produced by this tester; paste any of them above to see the full trace.

A longer Allow carves an exception out of a Disallow

robots.txt

User-agent: *
Disallow: /admin/
Allow: /admin/public/

/admin/public/list for GooglebotAllowed

Decided by Allow: /admin/public/. Both rules match; /admin/public/ is the longer path, so it wins.

A crawler-specific group replaces the * group entirely

robots.txt

User-agent: *
Disallow: /private/

User-agent: Googlebot
Disallow: /tmp/

/private/report for GooglebotAllowed

No rule matches. Googlebot obeys only its own group, which says nothing about /private/. Repeat shared rules in every named group.

Blocking GPTBot doesn't block other AI crawlers

robots.txt

User-agent: GPTBot
Disallow: /

User-agent: *
Allow: /

/blog/post for ClaudeBotAllowed

Decided by Allow: /. Each AI crawler needs its own User-agent line; ClaudeBot falls back to *.

$ stops matching when a query string follows

robots.txt

User-agent: *
Disallow: /*.pdf$

/files/guide.pdf?v=2 for GooglebotAllowed

No rule matches. The URL ends with ?v=2, not .pdf, so the anchored pattern doesn't match. Use Disallow: /*.pdf to cover both.

Paths are case-sensitive

robots.txt

User-agent: *
Disallow: /Admin/

/admin/settings for BingbotAllowed

No rule matches. /Admin/ and /admin/ are different paths. Match the case your URLs actually use.

Frequently asked questions

How do I test if a URL is blocked by robots.txt?

Paste or fetch the robots.txt, enter the URL or path, and pick a crawler. The tester shows allowed or blocked, the deciding rule, and the reasoning step by step.

Why is my page blocked when the file looks fine?

Usually group selection or precedence: the crawler matched a different group than you expected, or a broader wildcard rule matched. The trace shows which group and rule applied.

If a URL is allowed, will Google index it?

Not necessarily. Allowed means a compliant crawler may fetch it. Whether it is crawled, indexed, or ranked depends on the search engine.