Robots.txt Tester
A valid robots.txt can still block the wrong page. This tester answers the real question — is this URL allowed or blocked for this crawler? — and shows the exact rule that decided it, plus a step-by-step trace so the verdict is never a black box.
Why test individual URLs
Validation confirms a file is well-formed; it doesn't confirm your important pages are reachable. Real matching depends on group selection, longest-match precedence, and wildcard rules — easy to misjudge by reading the file. Testing applies the actual algorithm to a specific URL and crawler.
URL TesterTest a URLSee exactly why
Every result shows the verdict, the matched directive, and the reasoning: which user-agent group applied, which Allow/Disallow rules matched, and why one won. No guessing about precedence.
robots.txt:
User-agent: *
Disallow: /admin/
Allow: /admin/public/
URL: /admin/public/list
Crawler: Googlebot
→ ALLOWED by Allow: /admin/public/ (longer match wins)How to test a URL
- Provide your robots.txt — paste it or fetch it from a live domain.
- Enter the URL or path you want to check.
- Pick the crawler (Googlebot, GPTBot, Bingbot, and more).
- Read the verdict, the matched rule, and the trace.
Test the crawlers that matter
How do I test if a URL is blocked by robots.txt?
Provide your robots.txt, enter the URL, and choose a crawler. The tester applies the real matching rules and returns allowed or blocked, along with the exact directive that decided it.
Why is my page blocked when the file looks fine?
Usually because of group selection or precedence — a broad Disallow matched before a more specific Allow, or a wildcard caught more than expected. The tester's trace shows which rule applied and why.
Can I test different crawlers?
Yes. Rules can vary by user-agent, so you can test Googlebot, Bingbot, GPTBot, and others against the same URL to confirm each behaves the way you intend.
Is the robots.txt tester free?
Yes. Testing is free with no sign-up. You can paste a file or fetch it live, then test as many URLs and crawlers as you like.
URL Tester
See whether a URL is blocked or allowed for any crawler, with a rule trace.
Robots.txt Studio Editorial · Technical SEO & crawling
We build robots.txt tooling and parse thousands of real-world files. Guides are written by practitioners and reviewed against the Google and RFC 9309 specifications.