How to Find robots.txt
Finding a robots.txt file is simple once you know the rule: it's always at the same path on every site. Here's how to view yours, inspect a competitor's, and what a missing file means.
The one URL to remember
Append /robots.txt to the root of any domain and open it in a browser. The Robots Exclusion Protocol fixes this location — it's never anywhere else.
https://example.com/robots.txt
https://www.nytimes.com/robots.txt
https://github.com/robots.txtCheck the exact host
Finding your own robots.txt
- Visit https://yourdomain.com/robots.txt in a browser.
- If you see rules, that's your live file. Note whether it's a physical file or generated by your CMS/framework.
- If you get a 404, you don't have one — crawlers will assume “allow all.”
- For a full audit (validity, AI crawler coverage, sitemap), run it through the Analyzer.
Inspecting another website's robots.txt
Because robots.txt is public, you can read any site's file the same way. It's a quick way to learn how large sites handle crawl control, faceted navigation, and AI crawlers.
Paste any domain into the Analyzer to fetch and break down its file — which crawlers are blocked, how AI bots are handled, and any issues — without copy-pasting into a text editor.
What if /robots.txt returns 404?
A 404 means the site has no robots.txt. That's valid: crawlers treat a missing file as permission to crawl everything. If that's not what you want, create one.
Building your first file takes a minute in the Generator. If you're not sure what an existing file means, the Explainer translates it to plain English.
Common confusion
Looking in a subfolder
/blog/robots.txt is never used. Only the root file counts.
Checking the wrong subdomain
Your store on shop.example.com needs its own file; the root domain's doesn't apply.
Seeing an HTML page
If /robots.txt shows a styled page, the server is returning HTML, not a real robots.txt — crawlers will ignore it.
Where is robots.txt located?
Always at the root of the host, e.g. https://example.com/robots.txt. It cannot live in a subdirectory, and each subdomain has its own.
How do I find another website's robots.txt?
Add /robots.txt to its domain in your browser, or paste the domain into the Analyzer to fetch and audit it automatically.
What if /robots.txt returns a 404?
The site has no robots.txt, which crawlers treat as “allow everything.” Create one with the Generator if you want to control crawling or declare a sitemap.
Can I view robots.txt on a phone?
Yes — it's just a web page. Open https://domain.com/robots.txt in any mobile browser.
Robots.txt Analyzer
Fetch and audit any site's live robots.txt in one report.
Robots.txt Studio Editorial · Technical SEO & crawling
We build and maintain the robots.txt parser, validator, and tester behind these tools. Guides are checked against Google Search Central documentation and RFC 9309, and their worked examples are verified by the same engine.