Blocking SEMRUSHBOT from website

Blocking SEMrush bot might be necessary if you are seeing a sudden increase in bandwidth, especially if your Google Analytics isn't showing it.
This is one of our smaller recent projects - November 10, 2018

Blocking SEMrush Bot Attacks

I noticed on one of my client sites that the bandwidth usage had started to climb rapidly, despite the Google Analytics reports not showing a particularly large growth in traffic. Unusually high bandwidth usage may point to a site infected with malware, a site with particularly large and unoptimised images, or lots of good (or bad) bot activity.

There were some particularly large images on the site, so I optimised them using Imagify.io. This helped, but checking the bandwidth usage again a few days later showed that hits on the website were still unusually high.

Screenshot showing SEMRush bot usage on website

What is SEMrush Bot?

After a little research, using cpanel’s webstats utility, I found that the SEMRUSHBOT was generating around 60% of all site visits and around 75% of the bandwidth usage. SEMRush is a service that spiders the web looking for new and updated web data. It uses this data to help website owners compare against competitor sites, find keywords, look for issues, etc. However, this client site was not using the service, so the visits were of no use.

How to block SEMrush – robots.txt

SEMRush suggested using the following code in the robots.txt file to block their bot, saying that it might take 2 weeks for the bot to notice the change.

User-agent: SemrushBot
Disallow: /
User-agent: SemrushBot
Disallow: /
User-agent: SiteAuditBot
Disallow: /
User-agent: SemrushBot-BA
Disallow: /
User-agent: SemrushBot-SI
Disallow: /
User-agent: SemrushBot-SWA
Disallow: /
User-agent: SemrushBot-CT
Disallow: /
User-agent: SplitSignalBot
Disallow: /
User-agent: SemrushBot-COUB
Disallow: /

Should I block Semrush?

If you have registered your website with Semrush as a customer, to help improve your SEO and to have a better understanding of your keywords and competition, then you really need to allow the Semrush bot access to your website. Better, though, would be to change your Semrush settings to crawl your website as Googlebot instead. You can then block the Semrush bot with robots.txt as described above.

Other bots to block in your htaccess file

Whilst you’re editing your robots.txt to block SEMrushbot you might also want to consider adding the following to block other sites that crawl your website too often…

User-agent: MJ12bot
Disallow: /

Having trouble identifying what’s causing your high bandwidth usage? Are you struggling to get the most from SEMrush? We can help you if you complete the following form…

  • Here's how we will use the information you provide.
  • This field is for validation purposes and should be left unchanged.

Need help with your SEO?

If you are trying to make sense of SEMrush, or are generally trying to improve your organic search traffic, we can help.

1 Comment

  1. Tim

    Thanks for publishing this. Just this month I started having bandwidth issues that my host was telling me would require a new plan for, but I couldn’t figure out why as everything else, like the number of visitors was pretty much the same as usual.

    I stumbled across this page in searching for help, and sure enough, I found SemrushBot having used almost 3 GB of bandwidth this month. That’s ridiculous! How is that different from a denial of service attack?

    I used to think Googlebot was excessive, but it has nothing on SemrushBot.

    Thanks for sharing this, I appreciate it.

    Reply

Submit a Comment

Your email address will not be published. Required fields are marked *

A few of our other recent projects…

Share This
Martin Jarvis

Martin Jarvis

I typically reply within an hour

We're here Monday - Friday during UK working hours. Drop us an email and we'll get back to you.

Martin Jarvis
Hi. It's Martin here. How can I help you?
Start Chat with:
chat