Preserve SourceIP address in Kubernetes and distribute the load

user_23388 0
Viewed 365

In a multiple node cluster we want to expose a service handling UDP traffic. There are two requirements:

  1. We want the service to be backed up by multiple pods (possibly running on different nodes) in order to scale horizontally.
  2. The service needs the UDP source IP address of the client (i.e., should use DNAT instead of SNAT)

Is that possible?

We currently use a NodePort service with externalTrafficPolicy: local. This forces DNAT but only the pod running on the requested node is receiving the traffic. There doesn't seem to be a way to spread the load over multiple pods on multiple mnodes.

I already looked at this Kubernetes tutorial and also this article here.

kubernetes-ingress kubernetes networking
1 Answers

The Problem

I feel like there is a need for some explanation before facing the actual issue(s) in order to understand why things do not work as expected:

Usually what happens when using NodePort is that you expose a port on every node in your cluster. When making a call to node1:port the traffic will then (same as with a ClusterIP type) be forwarded to one Pod that matches the selector, regardless of that Pod being on node1 or another node.

Now comes the tricky part. When using externalTrafficPolicy: Local, packages that arrive on a node that does not have a Pod on it will be dropped. Perhaps the following illustration explains the behavior in a more understandable way.

NodePort with default externalTrafficPolicy: Cluster:

package --> node1 --> forwards to random pod on any node (node1 OR node2 OR ... nodeX)

NodePort with externalTrafficPolicy: Local:

package --> node1 --> forwards to pod on node1 (if pod exists on node1)

package --> node1 --> drops package (if there is no pod on node1)

So in essence to be able to properly distribute the load when using externalTrafficPolicy: Local two main issues need to be addressed:

  1. There has to be a Pod running on every node in order for packages not to be dropped
  2. The client has to send packages to multiple nodes in order for the load to be distributed

The solution

The first issue can be resolved rather easily by using a DaemonSet. It will ensure that one instance of the Pod runs on every node in the cluster.

Alternatively one could also use a simple Deployment, manage the replicas manually and ensure proper distribution across the nodes by using podAntiAffinity. This approach would take more effort to maintain since replicas must be adjusted manually but can be useful if you want to have more than just 1 Pod on each node.

Now for the second issue. The easiest solution would be to let the client implement logic on his part and send requests to all the nodes in a round robin principle, however, that is not a very practical and/or realistic way of doing it.

Usually when using NodePort there is still a load balancer of some kind in front of it to distribute the load (not taking about the Kubernetes service type LoadBalancer here). This may seem redundant since by default NodePort will distribute the traffic across all the Pods anyways, however, the node that gets requested still gets the traffic and then another hop happens. Furthermore if only the same node is addressed at all time, once that node goes down (for whatever reason) traffic will never reach any of the Pods anyways. So for those (and many other reasons) a load balancer should always be used in combination with NodePort. To solve the issue simply configure the load balancer to preserve the source IP of the original client.

Furthermore, depending on what cloud you are running on, there is a chance of you being able to configure a service type LoadBalancer instead of NodePort (which basically is a NodePort service + a load balancer in front of it as described above) , configure it with externalTrafficPolicy: Local and address the first issue as described earlier and you achieved what you wanted to do.

user_159635730
Related