Blocking SEMrush Bot Attacks
I noticed on one of my client sites that the bandwidth usage had started to climb rapidly, despite the Google Analytics reports not showing a particularly large growth in traffic. Unusually high bandwidth usage may point to a site infected with malware, a site with particularly large and unoptimised images, or lots of good (or bad) bot activity.
There were some particularly large images on the site, so I optimised them using Imagify.io. This helped, but checking the bandwidth usage again a few days later showed that hits on the website were still unusually high.
What is SEMrush Bot?
After a little research, using cpanel’s webstats utility, I found that the SEMRUSHBOT was generating around 60% of all site visits and around 75% of the bandwidth usage. SEMRush is a service that spiders the web looking for new and updated web data. It uses this data to help website owners compare against competitor sites, find keywords, look for issues, etc. However, this client site was not using the service, so the visits were of no use.
How to block SEMrush – robots.txt
SEMRush suggested using the following code in the robots.txt file to block their bot, saying that it might take 2 weeks for the bot to notice the change.
Should I block Semrush?
If you have registered your website with Semrush as a customer, to help improve your SEO and to have a better understanding of your keywords and competition, then you really need to allow the Semrush bot access to your website. Better, though, would be to change your Semrush settings to crawl your website as Googlebot instead. You can then block the Semrush bot with robots.txt as described above.
Other bots to block in your htaccess file
Whilst you’re editing your robots.txt to block SEMrushbot you might also want to consider adding the following to block other sites that crawl your website too often…
Having trouble identifying what’s causing your high bandwidth usage? Are you struggling to get the most from SEMrush? We can help you if you complete the following form…
Need help with your SEO?
If you are trying to make sense of SEMrush, or are generally trying to improve your organic search traffic, we can help.