Kubernetes Inter-Node traffic is not working properly

user_7630087 0
Viewed 58

lately I am configuring a k8s cluster composed of 3 nodes(master, worker1 and worker2) that will host an UDP application(8 replicas of it). Everything is done and the cluster is working very well but there is only one problem.

Basically there is a Deployment which describes the Pod and it looks like:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: <name>
  labels:
    app: <app_name>
spec:
  replicas: 8
  selector:
    matchLabels:
      app: <app_name>
  template:
    metadata:
      labels:
        app: <app_name>
    spec:
      containers:
      - name: <name>
        image: <image>
        ports:
          - containerPort: 6000
            protocol: UDP

There is also a Service which is used to access to the UDP application:

apiVersion: v1
kind: Service
metadata:
  name: <service_name>
  labels:
    app: <app_name>
spec:
  type: NodePort
  ports:
  - port: 6000
    protocol: UDP
    nodePort: 30080
  selector:
    app: <app_name>

When i try to access to the service 2 different scenarios may occur:

  • The request is assigned to a POD that is in the same node that received the request
  • The request is assigned to a POD that is in the other node

In the second case the request arrives correctly to the POD but with a source IP which ends by 0 (for example 10.244.1.0) so the response will never be delivered correctly.

I can't figure it out, I really tried everything but this problem still remains. In this moment to make the cluster working properly i added externalTrafficPolicy: Local and internalTrafficPolicy: Local to the Service in this way the requests will remain locally so when a request is sent to worker1 it will be assigned to a Pod which is running on worker1, the same for the worker2.

Do you have any ideas about the problem? Thanks to everyone.

kubernetes-pod kubernetes-ingress kubernetes networking
1 Answers

Have you confirmed that the response is not delivered correctly for your second scenario? The source IP address in that case should be the one of the node where the request first arrived.

I am under the impression that you are assuming that since the IP address ends in 0 this is necessarily a network address, and that could be a wrong assumption, as it depends on the Netmask configured for the Subnetwork where the nodes are allocated; for example, if the nodes are in the Subnet 10.244.0.0/23, then the network address is 10.244.0.0, and 10.244.1.0 is just another usable address that can be assigned to a node.

Now, if your application needs to preserve the client's IP address, then that could be an issue since, by default, the source IP seen in the target container is not the original source IP of the client. In this case, additionally to configuring the externalTrafficPolicy as Local, you would need to configure a healthCheckNodePort as specified in the Preserving the client source IP documentation.

user_169223430
Related