Magento robots.txt
Magento's biggest crawl problem is faceted navigation: filters and sort parameters multiply a handful of category pages into thousands of near-duplicate URLs. A well-tuned robots.txt is the first line of defense. Here's a practical Magento 2 setup.
Where to set it in Magento
Magento 2 manages robots.txt from the admin: Content → Design → Configuration → edit your store view → Search Engine Robots. You can pick a preset and add custom instructions. On multi-store setups, set it per store view and per domain — each host needs its own correct file.
Set the base URL first
Recommended Magento robots.txt
Block the technical directories and the parameter URLs that faceted navigation generates, keep product and category pages crawlable, and declare your sitemap. Tune the parameter list to your store's actual filters.
User-agent: *
Disallow: /checkout/
Disallow: /customer/
Disallow: /cart/
Disallow: /catalogsearch/
Disallow: /*?p=
Disallow: /*?price=
Disallow: /*?color=
Disallow: /*?size=
Disallow: /*?dir=
Disallow: /*?order=
Sitemap: https://example.com/sitemap.xmlFaceted navigation: the core issue
Every filter combination (color + size + price + sort) is a distinct URL. Left unchecked, crawlers spend your entire budget on these instead of real products. Disallowing the parameter patterns is the blunt fix; canonical tags and proper parameter handling are the precise one. Use both.
Don't block what you want ranked
Common Magento mistakes
Blocking /media/, /static/, or /pub/static/
These hold CSS, JS, and product images. Blocking them breaks rendering and image search. Keep them crawlable.
Forgetting per-store-view files
Multi-site Magento serves different robots.txt per domain. Configure each one.
Relying only on robots.txt for facets
Combine with canonical tags and URL parameter handling for clean indexing.
Shipping the default sample
Magento's example includes paths that may not match your setup. Review every line.
Verify the live file
After saving, flush Magento's cache and fetch the live robots.txt — full-page cache and Varnish can serve a stale copy. Then audit it for unintended blocks.
Robots.txt AnalyzerAnalyze a siteWhere is the robots.txt file in Magento 2?
Magento 2 manages it from the admin under Content → Design → Configuration → Search Engine Robots. It's served at /robots.txt per store view, so multi-domain stores configure each one separately.
How do I stop Magento faceted navigation wasting crawl budget?
Disallow the filter and sort parameter patterns (e.g. /*?color=, /*?p=, /*?order=) in robots.txt, and pair that with canonical tags and parameter handling so crawlers focus on real product and category pages.
Should I block /media/ or /static/ in Magento?
No. Those directories hold CSS, JavaScript, and product images that Google needs to render pages and power image search. Blocking them can hurt rankings.
What should a Magento robots.txt include?
Disallow checkout, customer, cart, and catalog search; disallow the faceted-navigation parameters your store uses; keep assets crawlable; and declare your sitemap with the canonical https URL.
Robots.txt Validator
Catch syntax errors and best-practice issues, with a health score.
Robots.txt Studio Editorial · Technical SEO & crawling
We build robots.txt tooling and parse thousands of real-world files. Guides are written by practitioners and reviewed against the Google and RFC 9309 specifications.