Dealing With Bad Bots at The Firewall Level

This weekend I noticed a bot from Attributor Corp was pounding my server. So I went ahead and disallowed the bot by adding the following code

User-agent: attributor.com
Disallow: /
to the robots.txt file. I checked back a little later and noticed the server was still being crawled heavily by this bot, leaving me to believe that this bot was bad behaving bot and ignoring my robots.txt file. So I decided I wanted to block them at the firewall level. So I went ahead and looked up their IP range and quickly found it was from 64.41.145.0/24 . I have a startup script that I run on the server that does certain tasks for me such as adding rules to the firewall. I opened it up and added the following to it:
ipfw add deny ip from 64.41.145.0/24 to any

 

This seems to have taken care of the problem. I also added Cyveillance and Twiceler, which also seem to be bad behaving. It might be overkill but nothing annoys me more than bad bots. So now my startup script now looks like this:

#block attributor corp
ipfw add deny ip from 64.41.145.0/24 to any

#block cyveillance
ipfw add deny ip from 63.148.99.224/27 to any
ipfw add deny ip from 63.148.99.0/24 to any
ipfw add deny ip from 65.118.41.192/27 to any
ipfw add deny ip from 65.118.41.0/24 to any 
ipfw add deny ip from 65.222.185.0/24 to any
ipfw add deny ip from 65.222.176.0/24 to any
ipfw add deny ip from 38.112.21.0/24 to any  
ipfw add deny ip from 38.118.25.0/24 to any  
ipfw add deny ip from 38.118.42.0/24 to any  
ipfw add deny ip from 38.118.25.56/29 to any
ipfw add deny ip from 38.118.42.32/29 to any
ipfw add deny ip from 207.87.178.0/24 to any
ipfw add deny ip from 216.32.64.0/24 to any
ipfw add deny ip from 63.100.163.0/24 to any
ipfw add deny ip from 151.173.221.0/24 to any
ipfw add deny ip from 68.48.24.0/24 to any
ipfw add deny ip from 4.35.201.0/24 to any
ipfw add deny ip from 63.148.99.224/27 to any
ipfw add deny ip from 38.100.41.64/26 to any

ipfw add deny ip from 65.118.41.192/27 to any
ipfw add deny ip from 65.205.32.0/24 to any

#block Cuill/Twiceler
ipfw add deny ip from 64.62.136.0/24 to any
ipfw add deny ip from 38.99.13.0/24 to any
ipfw add deny ip from 38.99.44.0/24 to any

 

If you've found this page then you're probably looking for a similar solution to the problem of bad bots ignoring your robots.txt file. I hope this helps you out.

Update Aug. 26, 2008

I'm using ipfw for this on my BSD variant server... but I've found that you can also deny these bots using iptables on other *nix variants. Here are the relevant commands:


iptables-A INPUT -s 64.41.145.0/24 -j DROP
iptables-A INPUT -s 63.148.99.224/27 -j DROP
iptables-A INPUT -s 63.148.99.0/24 -j DROP
iptables-A INPUT -s 65.118.41.192/27 -j DROP
iptables-A INPUT -s 65.118.41.0/24 -j DROP
iptables-A INPUT -s 65.222.185.0/24 -j DROP
iptables-A INPUT -s 65.222.176.0/24 -j DROP
iptables-A INPUT -s 38.112.21.0/24 -j DROP
iptables-A INPUT -s 38.118.25.0/24 -j DROP
iptables-A INPUT -s 38.118.42.0/24 -j DROP
iptables-A INPUT -s 38.118.25.56/29 -j DROP
iptables-A INPUT -s 38.118.42.32/29 -j DROP
iptables-A INPUT -s 207.87.178.0/24 -j DROP
iptables-A INPUT -s 216.32.64.0/24 -j DROP
iptables-A INPUT -s 63.100.163.0/24 -j DROP
iptables-A INPUT -s 151.173.221.0/24 -j DROP
iptables-A INPUT -s 68.48.24.0/24 -j DROP
iptables-A INPUT -s 4.35.201.0/24 -j DROP
iptables-A INPUT -s 63.148.99.224/27 -j DROP
iptables-A INPUT -s 38.100.41.64/26 -j DROP
iptables-A INPUT -s 65.118.41.192/27 -j DROP
iptables-A INPUT -s 65.205.32.0/24 -j DROP
iptables-A INPUT -s 64.62.136.0/24 -j DROP
iptables-A INPUT -s 38.99.13.0/24 -j DROP
iptables-A INPUT -s 38.99.44.0/24 -j DROP