I implemented a pretty simple but super effective rate limiting based on this blog post: https://www.nginx.com/blog/rate-limiting-nginx/
Basically:
limit_req_zone $binary_remote_addr zone=ip:10m rate=10r/s;
limit_req zone=ip burst=20 nodelay;
It works great. However, recently I tried Cloudflare, and this doesn't protect me anymore. I can bring down the site myself with a simple command of:
ab -k -c 1000 -n 10000 site.com/
What's happening?
ab -k -c 1000 -n 10000 site.com/ is running 1000 requests in parallel, until a total of 10 000 requests total have been done.
That's too brutal. It's likely that neither the client nor the server are tuned to handle thousands of connections over a few seconds.
Adjust the nginx configuration and do a gentle test ab -k -c 5 -n 500 site.com/
limit_req_zone $http_cf_connecting_ip zone=ip:10m rate=3r/s;
limit_req zone=ip;
limit_conn_status 429;
limit_req_status 429;
This configures nginx to return the standard status code 429 Too Many Requests when requests are rejected due to rate limiting.
nginx returns a 503 error by default (a bad default) meaning the application is failing, but it is not failing it is rate limited. It's important to configure status code appropriately to distinguish between server errors and rate limiting.
When behind cloudflare, nginx will not see the IP of the client but the IP of the cloudflare server. One might think that it breaks rate limiting by IP but it does not, well, just a bit.
When testing locally with ab, your test computer is only resolving a handful of cloudflare servers, and ab probably only uses the first IP. So no there aren't numerous clients IP, the rate limiting should work just fine.
When in production, there will be different clients accessing through different cloudflare servers. Still, there aren't that many cloudflare servers and clients in a geographic area will most likely resolve to the same cloudflare servers. So there will be a bunch of different IPs somewhat defeating the rate limiting, but probably not that many.
> nslookup mycloudflaresite.com
Name: mycloudflaresite.com
Addresses: 104.28.14.125
104.28.15.125
2606:4700:3037::681c:e7d
2606:4700:3036::681c:f7d
Cloudflare puts the original client IP in the CF-Connecting-IP header. It can also be in the X-Forwarded-For header or X-Real-Ip or True-Client-IP depending on settings and requests. See https://support.cloudflare.com/hc/en-us/articles/200170986-How-does-Cloudflare-handle-HTTP-Request-headers-
Hence the above configuration does rate limiting by client IP using the CF-Connecting-IP header. The nginx variable $binary_remote_addr would be the cloudflare server IP.
X-Forwarded-For to rate limitThe X-Forwarded-For header can be controlled by the client. It shouldn't be used for rate limiting because it is trivial to circumvent.
Example with a client having the IP 100.11.22.33:
X-Forwarded-For header => Cloudflare sets X-Forwarded-For: 100.11.22.33 and CF-Connecting-IP: 100.11.22.33 on the request.X-Forwarded-For: dummyvalue header already set => CloudFlare sets X-Forwarded-For: dummyvalue,100.11.22.33 and CF-Connecting-IP: 100.11.22.33 on the request.As you can see, it's trivial for the client to put a random value per request and totally circumvent any rate limiting based on the X-Forwaded-For header.
I would suggest trying the following:
(not sure if your backend is PHP but you can readjust)
# Conection Limit, taking the best from:
# http://serverfault.com/questions/177461/how-to-rate-limit-in-nginx-but-including-excluding-certain-ip-addresses
# http://gadelkareem.com/2015/01/18/limiting-connections-requests-wordpress-nginx/
geo $whitelist {
default 0;
# CIDR in the list below are not limited ( 1 )
127.0.0.1/32 1;
}
map $whitelist $limit {
0 $binary_remote_addr;
1 "";
}
# The directives below limit concurrent connections from a
# non-whitelisted IP address to five
limit_conn_zone $limit zone=conlimit:30m;
limit_conn_zone $limit zone=conlimit_php:30m;
limit_conn conlimit 40;
limit_conn_log_level warn; # logging level when threshold exceeded
limit_conn_status 503; # the error code to return
# Limit Req Non - PHP
limit_req_zone $limit zone=reqlimit:30m rate=5r/s;
limit_req zone=reqlimit burst=10;
# Limit Req PHP
limit_req_zone $limit zone=reqlimit_php:30m rate=1r/s;
limit_req_log_level warn;
limit_req_status 503;
# Location VirtualHost
limit_req zone=reqlimit_php burst=4;
limit_conn conlimit_php 10;
And for CF-Connecting-IP, you can whitelist the Cloudflare IPs via:
#!/bin/bash
cloudflare_real_ip_conf='/usr/local/nginx/conf/cloudflare_ip.conf'
echo "#Cloudflare" > ${cloudflare_real_ip_conf};
for i in `curl -sk https://www.cloudflare.com/ips-v4`; do
echo "set_real_ip_from $i;" >> ${cloudflare_real_ip_conf};
done
for i in `curl -sk https://www.cloudflare.com/ips-v6`; do
echo "set_real_ip_from $i;" >> ${cloudflare_real_ip_conf};
done
echo "" >> ${cloudflare_real_ip_conf};
echo "# use any of the following two" >> ${cloudflare_real_ip_conf};
echo "real_ip_header CF-Connecting-IP;" >> ${cloudflare_real_ip_conf};
echo "#real_ip_header X-Forwarded-For;" >> ${cloudflare_real_ip_conf};
which will produce the CF whitelist (currently):
#Cloudflare
set_real_ip_from 173.245.48.0/20;
set_real_ip_from 103.21.244.0/22;
set_real_ip_from 103.22.200.0/22;
set_real_ip_from 103.31.4.0/22;
set_real_ip_from 141.101.64.0/18;
set_real_ip_from 108.162.192.0/18;
set_real_ip_from 190.93.240.0/20;
set_real_ip_from 188.114.96.0/20;
set_real_ip_from 197.234.240.0/22;
set_real_ip_from 198.41.128.0/17;
set_real_ip_from 162.158.0.0/15;
set_real_ip_from 104.16.0.0/12;
set_real_ip_from 172.64.0.0/13;
set_real_ip_from 131.0.72.0/22;
set_real_ip_from 2400:cb00::/32;
set_real_ip_from 2606:4700::/32;
set_real_ip_from 2803:f800::/32;
set_real_ip_from 2405:b500::/32;
set_real_ip_from 2405:8100::/32;
set_real_ip_from 2a06:98c0::/29;
set_real_ip_from 2c0f:f248::/32;
# use any of the following two
real_ip_header CF-Connecting-IP;
#real_ip_header X-Forwarded-For;
You can add below to you nginx config
real_ip_header CF-Connecting-IP;
See the below article for more details