Block Bad Bots Using .htaccess

It is astonishing to think that 2012 was the year that traffic generated by automated bots and spiders on the internet outgrew human traffic. Since then, bots and spiders have only increased in their virility and use.

Whilst many bots are 'good' bots that do things like look through your site to calculate your position in search engine indexes or help you find the cheapest deal on your car insurance, many others are far less benign and more hostile: probing your website for security flaws or systems they can exploit.

If you have ever had a WordPress website hacked (and really ‑ who hasn't), it would inevitably have been a bot, rather than an actual human mortal, doing the hacking.

If you spot bots in your server history that are behaving oddly; maybe trying to access different variations of admin or wp URLs on your site in the hope of finding a log‑in, or just simply clogging your website down with irrelevant traffic, there are steps you can take to banish them.

The important thing to bear in mind here is that these solutions rely on the User-Agent string, which the client can change freely. They may reduce traffic from a small, stable nuisance bot, but they are not an authentication, authorisation, or abuse‑prevention boundary.

robots.txt

The first step is inside robots.txt. This is a file in the root of your domain which politely tells bots whether you would really rather they gave your website a skip. It's a little like those 'no soliciting' stickers your grandparents have on their front door, and about as useful.

Nevertheless, here is an example of how to block a bot called 'CuteStat' in your robots.txt file:

User-agent: CuteStatDisallow: /

This simply says "If you are a bot called CuteStat, you are not allowed anywhere beneath the root of this domain". This is actually a genuine example ‑ CuteStat are incredibly annoying, but at least they do pay attention to this disallow...

You can see my robots.txt file here if you are interested. As I mentioned though, these are really only useful to 'good' robots who actually pay any attention to the robots.txt standard. Like those religious doorstep visitors who you simply cannot stop interrupting your tea time, bad robots, and bad robot developers will simply ignore it.

.htaccess

Your second option is a little more technical, and will only work if you're on an Apache server with access to .htaccess and the required modules. Here, you can inspect a visitor's self‑reported User-Agent string and reject a request that matches a known nuisance pattern:

SetEnvIfNoCase User-Agent .*ahrefsbot.* bad_botSetEnvIfNoCase User-Agent .*dotbot.* bad_bot<Limit GET POST HEAD>  Order Allow, Deny  Allow from all  Deny from env=bad_bot</Limit>

Here, we set a variable called bad_bot when the User-Agent contains a listed string, then denies the request when that variable is true. A bot can bypass this by sending a different value, so the rule is a coarse filter rather than proof that a request is benign or hostile.

I've left a couple of bot examples in the code block above. Keep any such list short and evidence‑based; a large, reactive User-Agent list is brittle and belongs in a more appropriate edge or abuse‑control layer. A single additional pattern would look like this:

SetEnvIfNoCase User-Agent .*Go-http-client* bad_bot

One quick word of warning: every directive in your .htaccess file is evaluated as Apache handles requests, so a long rule set adds work and maintenance risk. Test the configuration before reloading it, watch legitimate traffic for false positives, and measure the effect rather than assuming it is negligible.

For sustained attacks or meaningful abuse, enforce controls at a higher layer: use a firewall or edge service, rate limits, behavioural signals, and trusted authentication and authorisation at the origin. Keep the User-Agent rule only as a limited nuisance filter.