SemrushBot in robots.txt
SemrushBot is Semrush's crawler. Here's exactly what it does, whether it respects robots.txt, and the rules to control it. To apply a policy in one click, use the AI Crawler Manager.
What SemrushBot is
SemrushBot is the crawler for Semrush, an all-in-one SEO and marketing platform. It collects backlink, keyword, and site-audit data. Like other SEO crawlers, it has no influence on your actual search rankings.
| Property | Value |
|---|---|
| User-agent | SemrushBot |
| Operator | Semrush |
| Category | SEO crawlers |
| Honors robots.txt | Yes |
| Affects search ranking | No |
Official documentation: Semrush crawler docs.
What SemrushBot does
- Crawls pages and links to build Semrush's backlink and competitive-research databases.
- Runs several named sub-bots (e.g. for site audit and backlink discovery).
Why site owners care
- Blocking SemrushBot limits competitor and backlink visibility in Semrush.
- It can add crawl load on large sites without sending traffic.
- No effect on Google or Bing rankings.
How to allow or block SemrushBot
Add a group targeting the SemrushBot user-agent. Disallow: / blocks it from your whole site; an empty Disallow: allows it.
User-agent: SemrushBot
Disallow: /User-agent: SemrushBot
Disallow:No effect on search ranking
How to verify SemrushBot
Verify via reverse DNS to a semrush.com / semrush.net host, or Semrush's published IP ranges.
Does it honor robots.txt?
Recommendation
Recommended: Allow
Does SemrushBot affect my Google ranking?
No. SemrushBot only collects data for the Semrush platform. It is separate from search-engine crawlers, so blocking it has no effect on how you rank in Google or Bing.
Robots.txt Validator
Catch syntax errors and best-practice issues, with a health score.
Robots.txt Studio Editorial · Technical SEO & crawling
We build and maintain the robots.txt parser, validator, and tester behind these tools. Guides are checked against Google Search Central documentation and RFC 9309, and their worked examples are verified by the same engine.