How to Find robots.txt
Finding a robots.txt file is simple once you know the rule: it's always at the same path on every site. Here's how to view yours, inspect a competitor's, and what a missing file means.
The one URL to remember
Append /robots.txt to the root of any domain and open it in a browser. The Robots Exclusion Protocol fixes this location — it's never anywhere else.
https://example.com/robots.txt
https://www.nytimes.com/robots.txt
https://github.com/robots.txtCheck the exact host
Finding your own robots.txt
- Visit https://yourdomain.com/robots.txt in a browser.
- If you see rules, that's your live file. Note whether it's a physical file or generated by your CMS/framework.
- If you get a 404, you don't have one — crawlers will assume “allow all.”
- For a full audit (validity, AI crawler coverage, sitemap), run it through the Analyzer.
Inspecting another website's robots.txt
Because robots.txt is public, you can read any site's file the same way. It's a quick way to learn how large sites handle crawl control, faceted navigation, and AI crawlers.
Paste any domain into the Analyzer to fetch and break down its file — which crawlers are blocked, how AI bots are handled, and any issues — without copy-pasting into a text editor.
What if /robots.txt returns 404?
A 404 means the site has no robots.txt. That's valid: crawlers treat a missing file as permission to crawl everything. If that's not what you want, create one.
Building your first file takes a minute in the Generator. If you're not sure what an existing file means, the Explainer translates it to plain English.
Common confusion
Looking in a subfolder
/blog/robots.txt is never used. Only the root file counts.
Checking the wrong subdomain
Your store on shop.example.com needs its own file; the root domain's doesn't apply.
Seeing an HTML page
If /robots.txt shows a styled page, the server is returning HTML, not a real robots.txt — crawlers will ignore it.
Where is robots.txt located?
Always at the root of the host, e.g. https://example.com/robots.txt. It cannot live in a subdirectory, and each subdomain has its own.
How do I find another website's robots.txt?
Add /robots.txt to its domain in your browser, or paste the domain into the Analyzer to fetch and audit it automatically.
What if /robots.txt returns a 404?
The site has no robots.txt, which crawlers treat as “allow everything.” Create one with the Generator if you want to control crawling or declare a sitemap.
Can I view robots.txt on a phone?
Yes — it's just a web page. Open https://domain.com/robots.txt in any mobile browser.
Robots.txt Analyzer
Fetch and audit any site's live robots.txt in one report.
Robots.txt Studio Editorial · Technical SEO & crawling
We build robots.txt tooling and parse thousands of real-world files. Guides are written by practitioners and reviewed against the Google and RFC 9309 specifications.