Blocking SEMRUSHBOT from website

Blocking SEMrush bot might be necessary if you are seeing a sudden increase in bandwidth, especially if your Google Analytics isn't showing it.
This is one of our smaller recent projects - November 10, 2018

Blocking SEMrush Bot

I noticed on one of my client sites that the bandwidth usage had started to climb rapidly, despite the Google Analytics reports not showing a particularly large growth in traffic. Unusually high bandwidth usage may point to a site infected with malware, a site with particularly large and unoptimised images, or lots of good (or bad) bot activity.

There were some particularly large images on the site, and so I optimised them using This helped, but checking the bandwidth usage again a few days later showed that hits on the website were still unusually high.

Screenshot showing SEMRush bot usage on website

What is SEMrush Bot?

After a little research, using cpanel’s webstats utility, I found that the SEMRUSHBOT was generating around 60% of all site visits, and around 75% of the bandwidth usage. SEMRush is a service that spiders the web looking for new and updated web data. It uses this data to help website owners compare against competitor sites, find keywords, look for issues etc. However, this client site was not using the service, and so the visits were of no use.

How to block SEMrush

SEMRush suggested using the following code in the robots.txt file to block their bot, saying that it might take 2 weeks for the bot to notice the change.

User-agent: SemrushBot
Disallow: /

Having trouble identifying what’s causing your high bandwidth usage? Are you struggling to get the most from SEMrush? We can help you if you complete the following form…

  • Here's how we will use the information you provide.
  • This field is for validation purposes and should be left unchanged.

Need help with your SEO?

If you are trying to make sense of SEMrush, or are generally trying to improve your organic search traffic, we can help.

1 Comment

  1. Tim

    Thanks for publishing this. Just this month I started having bandwidth issues that my host was telling me would require a new plan for, but I couldn’t figure out why as everything else, like the number of visitors was pretty much the same as usual.

    I stumbled across this page in searching for help, and sure enough, I found SemrushBot having used almost 3 GB of bandwidth this month. That’s ridiculous! How is that different from a denial of service attack?

    I used to think Googlebot was excessive, but it has nothing on SemrushBot.

    Thanks for sharing this, I appreciate it.


Submit a Comment

Your email address will not be published. Required fields are marked *

A few of our other recent projects…

Share This