Blocking SEMRUSHBOT from website

I noticed on one of my client sites that the bandwidth usage had started to climb rapidly, despite the Google Analytics reports not showing a particularly large growth in traffic. Unusually high bandwidth usage may point to a site infected with malware, a site with particularly large and unoptimised images, or lots of good (or bad) bot activity.

There were some particularly large images on the site, and so I optimised them using Imagify.io. This helped, but checking the bandwidth usage again a few days later showed that hits on the website were still unusually high.

Screenshot showing SEMRush bot usage on website

After a little research, using cpanel’s webstats utility, I found that the SEMRUSHBOT was generating around 60% of all site visits, and around 75% of the bandwidth usage. SEMRush is a service that spiders the web looking for new and updated web data. It uses this data to help website owners compare against competitor sites, find keywords, look for issues etc. However, this client site was not using the service, and so the visits were of no use.

SEMRush suggested using the following code in the robots.txt file to block their bot, saying that it might take 2 weeks for the bot to notice the change.

User-agent: SemrushBot
Disallow: /

Having trouble identifying what’s causing your high bandwidth usage? We can help you if you complete the following form…

  • Here's how we will use the information you provide.
  • This field is for validation purposes and should be left unchanged.
No comments yet.

Leave a Reply