WordPress robots.txt
WordPress generates a virtual robots.txt by default, and most SEO plugins let you edit it. The defaults are mostly fine — but a few stubborn mistakes (blocking /wp-content/, breaking admin-ajax) still cause real damage. Here's the version that works.
Virtual vs static robots.txt
If no physical robots.txt exists in your site root, WordPress serves a virtual one dynamically. The moment you upload a real robots.txt file (or an SEO plugin writes one), that static file takes over. Know which one you're editing — editing the virtual file's filters in code has no effect once a static file exists.
Plugins usually win
Recommended WordPress robots.txt
Block the admin area, but explicitly allow admin-ajax.php (themes and plugins call it on the front end) and never block wp-content — Google needs your CSS, JS, and images to render and rank pages.
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/wp-sitemap.xmlCommon WordPress mistakes
Disallow: /wp-content/
This blocks your CSS, JS, and images. Google renders pages — broken rendering can tank rankings. Never block it.
Blocking admin-ajax.php
Many front-end features (lazy load, filters, add-to-cart) use it. Always Allow: /wp-admin/admin-ajax.php.
Disallowing /wp-includes/
Old advice that blocks core scripts needed for rendering. Drop it.
Leaving the staging block live
WordPress under development often ships Disallow: / — confirm production doesn't.
Using Disallow to hide /wp-admin/ for security
robots.txt is public and advertises the path. Security comes from authentication, not Disallow.
AI crawlers on WordPress
Publishers and bloggers increasingly opt out of AI training. WordPress doesn't do this for you — add the AI crawler groups yourself, above the catch-all group:
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
Disallow: /
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/wp-sitemap.xmlVerify before you trust it
WordPress caching and CDNs can serve a stale robots.txt. After editing, fetch the live file and confirm it matches what you intended — then spot-test a key URL.
Robots.txt AnalyzerAnalyze a siteWhat is the best robots.txt for WordPress?
Disallow /wp-admin/, explicitly Allow /wp-admin/admin-ajax.php, leave wp-content crawlable, and declare your sitemap. Add AI crawler groups if you want to opt out of model training.
Should I block wp-content in robots.txt?
No. wp-content holds your themes' CSS and JavaScript and your images. Google needs them to render and evaluate pages, so blocking wp-content can hurt rankings.
Where is the WordPress robots.txt file?
By default WordPress serves a virtual robots.txt at /robots.txt with no physical file. If you or an SEO plugin create a real file in the site root, that static file replaces the virtual one.
How do I edit robots.txt in WordPress?
Use your SEO plugin (Yoast, Rank Math, All in One SEO) which provides a robots.txt editor, or upload a static robots.txt to the site root. Avoid doing both, since the static file overrides the plugin.
Robots.txt Validator
Catch syntax errors and best-practice issues, with a health score.
Robots.txt Studio Editorial · Technical SEO & crawling
We build robots.txt tooling and parse thousands of real-world files. Guides are written by practitioners and reviewed against the Google and RFC 9309 specifications.