Free robots.txt checker

Check your robots.txt: who can crawl your site?

Enter a domain to see which search engines, AI search crawlers, and AI training crawlers its robots.txt allows — with every error and a fix.

or paste a file into the validator · create a new one

Your crawler report

Unlock your full crawler report

Run a free analysis to reveal your search visibility score, AI-crawler access, and every robots.txt issue — no sign-up required.

Run free analysisTakes about 5 seconds

Everything you need to manage crawler access

Six powerful tools in one platform — no sign-up required to get started.

Analyze

Robots.txt Analyzer

Fetch any site's robots.txt and see which search engines and AI crawlers it lets in.

Open

Explainer

Read your robots.txt rules in plain English — no technical knowledge needed.

Open

Control

AI Crawler Manager

Decide AI training and AI search separately — GPTBot, ClaudeBot, OAI-SearchBot, Google-Extended and more.

Open

Generator

Build your robots.txt with a guided visual editor — no coding required.

Open

Verify

Validator

Check your robots.txt for syntax errors and warnings before they cause issues.

Open

URL Tester

Test whether a page is blocked or allowed for any crawler.

Open

AI visibility

AI training and AI search are separate choices

Opt out of training (GPTBot, ClaudeBot, Google-Extended) while still letting AI search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) cite and link you — or block both. robots.txt is a request to compliant crawlers, not access control. See the full AI crawler directory.

Manage AI access