SemrushBot in robots.txt
SemrushBot is Semrush's crawler. Here's exactly what it does, whether it respects robots.txt, and the rules to control it. To apply a policy in one click, use the AI Crawler Manager.
What SemrushBot is
SemrushBot is the crawler for Semrush, an all-in-one SEO and marketing platform. It collects backlink, keyword, and site-audit data. Like other SEO crawlers, it has no influence on your actual search rankings.
| Property | Value |
|---|---|
| User-agent | SemrushBot |
| Operator | Semrush |
| Category | SEO crawlers |
| Honors robots.txt | Yes |
| Affects search ranking | No |
Official documentation: Semrush crawler docs.
What SemrushBot does
- Crawls pages and links to build Semrush's backlink and competitive-research databases.
- Runs several named sub-bots (e.g. for site audit and backlink discovery).
Why site owners care
- Blocking SemrushBot limits competitor and backlink visibility in Semrush.
- It can add crawl load on large sites without sending traffic.
- No effect on Google or Bing rankings.
How to allow or block SemrushBot
Add a group targeting the SemrushBot user-agent. Disallow: / blocks it from your whole site; an empty Disallow: allows it.
User-agent: SemrushBot
Disallow: /User-agent: SemrushBot
Disallow:No effect on search ranking
How to verify SemrushBot
Verify via reverse DNS to a semrush.com / semrush.net host, or Semrush's published IP ranges.
Does it honor robots.txt?
Recommendation
Recommended: Allow
Does SemrushBot affect my Google ranking?
No. SemrushBot only collects data for the Semrush platform. It is separate from search-engine crawlers, so blocking it has no effect on how you rank in Google or Bing.
Robots.txt Validator
Catch syntax errors and best-practice issues, with a health score.
Robots.txt Studio Editorial · Technical SEO & crawling
We build robots.txt tooling and parse thousands of real-world files. Guides are written by practitioners and reviewed against the Google and RFC 9309 specifications.