Robots.txt StudioGet Started

Test a URL

Simulate how any crawler reads your robots.txt and understand exactly why a URL is blocked or allowed.

blocked

"GPTBot" is blocked from /admin/login by "Disallow: /".

Matched rule
User-agent: GPTBot
Disallow: /
Rule trace

  1. 1. Match user-agent

    Matched the most specific group, "User-agent: GPTBot", for "GPTBot".

  2. 2. Evaluate path rules

    1 rule(s) match "/admin/login": Disallow: /.

  3. 3. Apply precedence

    Longest match wins; Allow beats Disallow on ties. Winner: "Disallow: /" (line 6).

  4. 4. Decision

    Blocked by "Disallow: /".

More tools

Robots.txt Generator

Build a valid robots.txt from presets and crawler toggles — no syntax required.

Open

Robots.txt Validator

Catch syntax errors and best-practice issues, with a health score.

Open

Robots.txt Analyzer

Fetch and audit any site's live robots.txt in one report.

Open
Related guides

How Does robots.txt Work?

Group selection & precedence.

Open

How to Read robots.txt

Read rules line by line.

Open

robots.txt Syntax

Wildcards and anchors.

Open