Blocking SEMrush Bot
I noticed on one of my client sites that the bandwidth usage had started to climb rapidly, despite the Google Analytics reports not showing a particularly large growth in traffic. Unusually high bandwidth usage may point to a site infected with malware, a site with particularly large and unoptimised images, or lots of good (or bad) bot activity.
There were some particularly large images on the site, so I optimised them using Imagify.io. This helped, but checking the bandwidth usage again a few days later showed that hits on the website were still unusually high.
What is SEMrush Bot?
After a little research, using cpanel’s webstats utility, I found that the SEMRUSHBOT was generating around 60% of all site visits and around 75% of the bandwidth usage. SEMRush is a service that spiders the web looking for new and updated web data. It uses this data to help website owners compare against competitor sites, find keywords, look for issues, etc. However, this client site was not using the service, so the visits were of no use.
How to block SEMrush
SEMRush suggested using the following code in the robots.txt file to block their bot, saying that it might take 2 weeks for the bot to notice the change.
Having trouble identifying what’s causing your high bandwidth usage? Are you struggling to get the most from SEMrush? We can help you if you complete the following form…
Need help with your SEO?
If you are trying to make sense of SEMrush, or are generally trying to improve your organic search traffic, we can help.