Is it possible to make fail2ban ignore google?

user_8809335 0
Viewed 1400

I need to use fail2ban due to many attack attempts on my server, I also have filters that I had to activate/create to block attack attempts.

But now I'm pretty sure that some google ip ends up in the jail of my fail2ban...

I added some ip in the ignoreip directive in the jail.local file, but they are only the ones that I managed to identify as real google ip in my access.log (I also have many fake google)

It would be nice to be able to give a list of ip to ignore to fail2ban, but google does not release its ip list, google says: https://support.google.com/webmasters/answer/80553?hl=en

So the question is: is it possible to do a reverse dns to understand if an ip belongs to google and tell fail2ban to ignore it?

Can it be done via fail2ban? Do you need any external script? Could it be too heavy, long and tiring for the server?

fail2ban googlebot
2 Answers

yes, you can identify google bots using reverse IP lookup. all crawler bots will end with xxxxxx.google.com or xxxxxxx.googlebot.com

for e.g. crawl-203-208-60-1.googlebot.com

but it is not possible to identify in fail2ban, but you can whitelist the IP address once you know if its a Googlebot.

there are many ways to perform for reverse IP look.

you can use Python, Ruby or bash to find out. check the following article.

http://searchsignals.com/tutorials/reverse-dns-lookup/

there are websites that can find you reverse IP lookup.

https://dnschecker.org/reverse-dns.php

http://reverseip.domaintools.com/

if you can code in python, you easily dump reverse IP data in a file from a list of IP addresses.

user_73841470

Google does have a page about verifying GoogleBot addresses by doing a reverse-lookup on the IP address and verifying that it comes from a specific hostname (you'd then get the IP of that host, to double-check it comes back to the appropriate source IP).

There are also DNS TXT records that specify IP ranges for SPF (emails), Google Compute Cloud, and the wider Google IP addresses that can be used (many of which would be in use by GCP user's VMs and other services).

  • dig @8.8.8.8 +short TXT _spf.google.com
  • dig @8.8.8.8 +short TXT _cloud-netblocks.google.com
  • dig @8.8.8.8 +short TXT _cloud-netblocks.googleusercontent.com

The first query will return something like this:

"v=spf1 include:_netblocks.google.com include:_netblocks2.google.com include:_netblocks3.google.com ~all"

And you would then parse it to get the IP address ranges, or do a sub-query on the include:_netblocks.google.com etc to get other sets.

The information these records are not fixed, and can regularly change. (AWS publishes a .JSON file with several updates per week, for example).

I'm working on a system to automatically detect 'lying user-agents', with these, and some other techniques.

user_62160
Related