We're running an NGINX Ingress Controller as the 'front door' to our EKS cluster.
Our upstream apps need the client IP to be preserved, so I've had my ingress configmap configured to use the proxy protocol:
apiVersion: v1
kind: ConfigMap
metadata:
name: ingress-nginx-controller
namespace: ingress-nginx
data:
custom-http-errors: 404,503,502,500
ssl-redirect: "true"
ssl-protocols: "TLSv1.2"
ssl-ciphers: "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384:!ECDHE-RSA-AES256-SHA384:!ECDHE-RSA-AES128-SHA256"
use-proxy-protocol: "true"
proxy-real-ip-cidr: "0.0.0.0/0"
This sends the X-Forwarded-For header with the client IP to the upstream pods. This seemed like it was working well, but once our apps started to receive heavier traffic, our monitors would occasionally report connection timeouts when connecting to the apps on the cluster.
I was able reproduce the issue in a test environment using JMeter. Once I set use-proxy-protocol to false, the connection timeouts would no longer occur. I started to look into the use-proxy-protocol setting.
The ingress docs describe the use-proxy-protocol setting here
However the docs also mention the settings "enable-real-ip" and "forwarded-for-header"
At the link provided in the description for enable-real-ip, it says that I can set forwarded-for-header to value proxy_protocol.
Based on this, I've updated my Ingress configmap to:
apiVersion: v1
kind: ConfigMap
metadata:
name: ingress-nginx-controller
namespace: ingress-nginx-test
data:
custom-http-errors: 404,503,502,500
ssl-redirect: "true"
ssl-protocols: "TLSv1.2"
ssl-ciphers: "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384:!ECDHE-RSA-AES256-SHA384:!ECDHE-RSA-AES128-SHA256"
enable-real-ip: "true"
forwarded-for-header: "proxy_protocol"
proxy-real-ip-cidr: "0.0.0.0/0"
This configuration also properly sends the X-Forwarded-For header with the client IP to the upstream pods. However, it also seems to eliminate the connection timeout issues I was seeing. With this setup performance does not degrade anywhere near as badly as I ramp up the thread count in JMeter.
I would like to better understand the difference between these two configurations. I’d also like to know what is the best practice most widely adopted method of achieving this among Kubernetes shops since this is likely a common use-case.