How to expose tcp service in Kubernetes only for certain ip addresses?

Question

Nignx ingress provides a way to expose tcp or udp service: all you need is public NLB.

However this way tcp service will be exposed publicly: NLB does not support security groups or acl, also nginx-ingress does not have any way to filter traffic while proxying tcp or udp.

The only solution that comes to my mind is internal load balancer and separate non-k8s instance with haproxy or iptables, where I'll actually have restrictions based on source ip - and then forward/proxy requests to internal NLB.

Maybe there are another ways to solve this?

user_923620 · Answer

Do not use nginx-ingress for this. To get real IP inside nginx-ingress you have to set controller.service.externalTrafficPolicy: Local, which in its turn changes the way the nginx-ingress service is exposed - making it local to the nodes. See https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/#preserving-the-client-source-ip. This in its turn causes your nginx-ingress LoadBalancer to have unhealthy hosts which will create noise in your monitoring (opposite to NodePort where every node exposes the same port and healthy). Unless you run nginx-ingress as a DaemonSet or use other hacks, e.g. limit which nodes are added as a backends (mind scheduling, scaling), or move nginx-ingress to a separate set of nodes/subnet - IMO each of these is a lot of headache for such a simple problem. More on this problem: https://elsesiy.com/blog/kubernetes-client-source-ip-dilemma
Use plain Service type: LoadBalancer (classic ELB) which supports:
- Source ranges: https://aws.amazon.com/premiumsupport/knowledge-center/eks-cidr-ip-address-loadbalancer/
- service.beta.kubernetes.io/aws-load-balancer-extra-security-groups annotation in case you want to manage the source ranges from the outside.
In this case your traffic going like World -> ELB -> NodePort -> Service -> Pod, without Ingress.

user_3833773 · Answer

Yo can use the whitelist-source-range annotation for that. We've been using it successfully for a few use cases and it does the job well.

EDIT: I spoke too soon. Rereading your question and understanding your exact use case brought me to this issue, which clearly states these services cannot be whitelisted, and suggests solving this in the firewall level.

How to expose tcp service in Kubernetes only for certain ip addresses?

2 Answers