Our Methodology

Our analysis implements the Robots Exclusion Protocol as formalized in RFC 9309 (2022), aligned with Google Search Central's documented behavior. We wrote our own parser and matching engine rather than depending on a third-party library, so the logic is transparent and we control conformance. No large language model is involved in the core parsing, validation, testing, or scoring — identical input always yields identical output.

We parse the raw file into a canonical syntax tree that preserves fidelity: original line order, duplicate directives, and unknown directives are all retained rather than silently dropped. The parser is tolerant — malformed lines are recovered and surfaced as issues instead of crashing — so even a broken file produces a useful, line-referenced analysis.

• Recognized directives: User-agent, Allow, Disallow, Sitemap, Crawl-delay, Host, and comments.
• Groups are reconstructed from consecutive User-agent lines and the rules that follow them.
• Every directive keeps its source line number, so issues and explanations link back to exact lines.

To decide whether a URL is allowed or blocked for a crawler, we follow the RFC 9309 matching algorithm exactly:

1. Group selection — the crawler obeys the single group whose User-agent most specifically names it; if none matches, it falls back to the User-agent: * group.
2. Path evaluation — among the matching group's rules, the longest matching path wins (most specific rule).
3. Tie-breaking — when an Allow and a Disallow match the same length, Allow takes precedence.
4. Wildcards — * matches any sequence within a path, and $ anchors a match to the end of the URL.

The Validator runs a rule set across three dimensions — syntax correctness, AI crawler configuration, and best practices. Each finding carries a severity (error, warning, or recommendation), a line reference, and a plain-English description. The overall health score is decomposed into those three sub-scores using a single documented banding (80+ good, 60–79 moderate, below 60 poor) shared with the Analyzer's visibility score.

When you analyze a domain, our server fetches that domain's public robots.txt through a controlled route handler — not your browser — so we can manage redirects, timeouts, response size, and content type safely. We fetch only the robots.txt file at the host you enter.

• An 8-second timeout and a 512 KiB size cap protect against slow or oversized responses.
• A missing robots.txt (HTTP 404) is treated as 'all crawlers allowed' and reported as such — not as an error.
• Redirects, timeouts, network failures, and non-text content types each produce a distinct, explained result.
• The visibility score combines crawler access, restrictions, and syntax health into one explained number.

Our crawler database is a transparent config layer. Each crawler entry records its identity (name, operator, User-agent token), its category, a plain-English description of what it does, and a default recommendation. Classifications and compliance notes are compiled from each operator's published documentation and updated as that documentation changes.

A default recommendation (allow, block, or depends) reflects the typical site-owner choice for that crawler — it is informational, not advice tailored to your situation. Because User-agent strings can be spoofed, we document how to verify major crawlers by reverse DNS or published IP ranges on each crawler's directory page.

• Compliance varies by operator; we note known exceptions (for example, aggressive crawlers with a patchy record).
• robots.txt controls crawling, not indexing — a blocked URL can still be indexed if linked elsewhere.
• We analyze the file you provide or the public file we fetch; we cannot see server-level blocks or firewall rules.

We build robots.txt tooling and parse thousands of real-world files. Guides are written by practitioners and reviewed against the Google and RFC 9309 specifications. Reviewed against Google Search Central and RFC 9309.

If a crawler's behavior or documentation has changed, or you spot an error, please contact us — we keep classifications current.