Block Bad Bots Using .htaccess

It is astonishing to think that 2012 was the year that traffic generated by automated bots and spiders on the internet outgrew human traffic. Since then, bots and spiders have only increased in their virility and use.

Whilst many bots are 'good' bots that do things like look through your site to calculate your position in search engine indexes or help you find the cheapest deal on your car insurance, many others are far less benign and more hostile: probing your website for security flaws or systems they can exploit.

Automated tools can probe and exploit vulnerable WordPress sites, but a compromise does not tell us whether a bot or a person carried it out. Server logs and the affected files are better starting points for working out what happened.

If you spot bots in your server history that are behaving oddly; maybe trying to access different variations of admin or wp URLs on your site in the hope of finding a log‑in, or just simply clogging your website down with irrelevant traffic, there are steps you can take to banish them.

The important thing to bear in mind here is that these solutions rely on the User-Agent string, which the client can change freely. They may reduce traffic from a small, stable nuisance bot, but they are not an authentication, authorisation, or abuse‑prevention boundary.

robots.txt

The first step is inside robots.txt. This is a file in the root of your domain which politely tells bots whether you would really rather they gave your website a skip. It's a little like those 'no soliciting' stickers your grandparents have on their front door, and about as useful.

Nevertheless, here is an example of how to block a bot called 'CuteStat' in your robots.txt file:

User-agent: CuteStat
Disallow: /

This simply says "If you are a bot called CuteStat, you are not allowed anywhere beneath the root of this domain". This is actually a genuine example ‑ CuteStat are incredibly annoying, but at least they do pay attention to this disallow...

You can see my robots.txt file here if you are interested. As I mentioned though, these are really only useful to 'good' robots who actually pay any attention to the robots.txt standard. Like those religious doorstep visitors who you simply cannot stop interrupting your tea time, bad robots, and bad robot developers will simply ignore it.

.htaccess

Your second option is a little more technical, and will only work if you're on an Apache server with access to .htaccess and the required modules. Here, you can inspect a visitor's self‑reported User-Agent string and reject a request that matches a known nuisance pattern. The example below uses the older Apache 2.2 access‑control syntax. On Apache 2.4, it needs mod_access_compat as well as mod_setenvif, with the relevant directives permitted in .htaccess:

SetEnvIfNoCase User-Agent .*ahrefsbot.* bad_bot
SetEnvIfNoCase User-Agent .*dotbot.* bad_bot

<Limit GET POST HEAD>
  Order Allow,Deny
  Allow from all
  Deny from env=bad_bot
</Limit>

Here, we set bad_bot when the User-Agent contains a listed string, then deny matching GET, POST and HEAD requests. A bot can bypass this by sending a different value, so the rule is a coarse filter rather than proof that a request is benign or hostile. For an Apache 2.4 configuration, use the Require‑based access‑control approach from mod_authz_core instead of mixing old and new rules. With the same environment variable, Require all granted and Require not env bad_bot belong together inside RequireAll; retain the Limit block if only those methods should be filtered.

I've left a couple of bot examples in the code block above. Keep any such list short and evidence‑based; a large, reactive User-Agent list is brittle and belongs in a more appropriate edge or abuse‑control layer. A single additional pattern would look like this:

SetEnvIfNoCase User-Agent .*Go-http-client* bad_bot

One quick word of warning: every directive in your .htaccess file is evaluated as Apache handles requests, so a long rule set adds work and maintenance risk. Test the configuration before reloading it, watch legitimate traffic for false positives, and measure the effect rather than assuming it is negligible.

For sustained attacks or meaningful abuse, enforce controls at a higher layer: use a firewall or edge service, rate limits, behavioural signals, and trusted authentication and authorisation at the origin. Keep the User-Agent rule only as a limited nuisance filter.