Knowledgebase

How To Block Unwanted Bots From Your Site

If you run a site for a while, a time will come when you're overwhelmed with a bot, or a set of bots. Perhaps you notice a sudden spike in usage of your site, and you get a warning from your host. You investigate and find that a single unknown bot has been making hundreds of thousands of requests to your site, hogging your disk and CPU usage.

These threats are easily shut down once identified. In this tutorial, I'll show you how to block unwanted bots via the .htaccess file on your site.

Step 1: Get the Exact User Agent of the Bot

If you don't know which bots are hitting your site, you need to download the access logs from your cPanel, or check out the stats on your host's analytics program like Awstats. To check your logs, log into cPanel and scroll down to the "Visitor Stats" section as shown below and click "Raw Access Logs":

This will bring you to a page where you download all the access logs for the sites you control. Click the link and it'll download a ".gz" file onto your desktop. This is like a zipped file and needs to be extracted.

Most Windows installations these days come with a trial copy of the .gz extraction utility. However, yours could have expired and it's not worth it to buy a license. So I personally use 7-Zip - a free and open source gz file extractor that you can download from here.

Once you manage to extract the file onto the desktop, open it with an editor like Notepad++. Here, you can browse through each line to find out which bots are accessing your site. For example, here's an example of a Vietnamese bot called Coccocbot that accesses my site from time to time:

It's not actually a dangerous bot, and I haven't found it to be abusive. But I'll use it as an example for this article. You might also want to block it if you're not interested in the Vietnamese market!

Every bot identifies itself via a "User Agent" that looks like the one in the screenshot above:

coccocbot-web/1.0

Notice that all the user agent names in other entries look like this. So we're going to be blocking based on this string.

Step 2: Modify the .htaccess File on Your Site

The root directory of your site has a hidden file called ".htaccess". Depending on how you view the directory, it may or may not be visible. If you access the folder via cPanel, you can enable hidden files in the settings. If you're using an FTP program like WinSCP, you can enable it via preferences.

All WordPress sites have a ".htaccess" file in their root directory. So open it if you're using WordPress. If not, just create a blank file with the name ".htaccess" - notice the dot (.) in front of the filename. Now paste the following at the beginning:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} coccocbot-web/1.0 [NC]
RewriteRule .* - [F,L]

Like this:

Replace the text in bold with the user agent name you got in Step 1. Save your changes, and let's test it out!

Step 3: Testing the Changes with a Chrome Extension

You can test the changes to see if the new rule works. On Chrome, download the User-Agent Switcher extension. Now create a new user agent with the name of the bot you're trying to block as shown here:

The switch to that bot using the icon in the address bar and try and access your site. If you've set up the .htaccess rule properly, you should get a 403-forbidden error as shown here:

And that means your .htaccess rule is successful. Congratulations - you've blocked a disruptive bot through htaccess!

How To Block An IP Address In WordPress

There could be thousands of log in attempts and IP addresses that access your wp-login.php page in an attempt to DDoS your site.

The difference is that bots - even bad ones - generally give their name and it's easy to identify them. IP addresses on the other hand are a little more complicated. In this tutorial, I'll show you several methods to block IP addresses, including solutions you might not have thought you needed!

Solution 1: Most Common - Block from Comments

A very common problem is comment spam that somehow slips through your filters. I sometimes wake up to find my blog flooded with dozens of spam messages. It's not necessarily a strain on my server, but it's annoying to have to delete them over and over manually, and it creates a bad impression to visitors to the site who see the spam comments.

We can block these comments via IP. First, go to your comments section and see the IP name of the commenter like this:

Next, go to Settings -> Discussion:

Here, scroll down to the section labeled "Comment Blacklist" and add the IP address to the box as shown here:

Save your changes and you're done! Now all comments originating from that IP address will go straight to the trash can. It's a simple solution and it works great for most people.

Solution 2: Block IP Addresses in .htaccess

This is a bit more serious. Use this solution when it's more than just a dozen comments or hits on your site. This is for an IP address that actively tries to DDoS your site - whether accidentally, or on purpose.

Open the ".htaccess" file at the root of your site directory and paste in the following lines:

Order Allow,Deny
Deny from [IP Address]
Allow from all

Here, replace [IP Address] in bold with the IP address you need to block. Now sessions that originate from that IP address will get a 403 error as shown here:

This solution also puts the least strain on your server. Since no PHP files are being executed, there's barely any CPU or script usage.

Solution 3: Using a Plugin for IP Lists

The solution above with htaccess is great for one-off IP address bans. But if you find yourself doing it more often, then a better solution is to use a plugin. I personally use the iThemes security plugin. The free version is pretty good by itself and it comes with a bunch of other useful security tweaks. Once installed, navigate to the section on "Banned Users", and click "Enable Ban Lists":

Now enter the IP addresses you want to ban in the box below - one on each line. Save your changes and you're done!

Solution 4: Banning by Country with Cloudflare

If you notice a lot of spam coming from a specific country, you might want to consider a country specific IP ban list. However, this is far from trivial. IP block rules for countries can number hundreds of lines that change all the time. It's practically impossible to manually maintain a list of IP addresses by country. People have tried, and it usually turns out to be a mess. Don't do it.

A far better solution is to use a reverse proxy like Cloudflare that does the heavy work for you. If you're already using Cloudflare, then navigate to the "Firewall" section and scroll down to "Access Rules" as shown here:

Now just enter the country name you want to ban. You can create more than one blocking rule and block several countries if you want. It's much more efficient and effective than trying to do it by yourself.

So there you have it. A number of methods to implement IP bans on your site - from the simple comment moderation, to a full-blown country-level IP ban. I hope some of the information here helps keep you safe, and your site accessible!