ZMQ sockets do not work as expected on Kubernetes

Viewed 1288

Short summary: My ZMQ sockets do not receive (probably also send) messages when I deploy the code on kubernetes.

I have an application that involves multiple clients and servers. It is developed on node and uses ZeroMQ as communication layer. It works on my local machine, it works on docker and I am trying to deploy the application using kubernetes.

When deploying the application, the pods, the deployments and the kubernete's services launch. Apparently everything goes ok, but the clients send an initial message that never arrives to the server. The deployments are in the same namespace and I use Flannel as CNI. To the best of my knowledge, the cluster was properly initialized but the messages never arrive.

I read this question about ZMQ sockets having problems binding on kubernetes. I have tried playing with the ZMQ_CONNECT_TIMEOUT parameter but it does not do anything. Also, unlike the question I cite, my messages never arrive.

I could provide some code, but there is a lot of code and I don't think the application is the problem. I think I am missing something on the Kubernetes configuration since it's my firts time using it . Let me know if you need more information.

Edit 1. 12/01/2021

As @anemyte suggests I will try to provide a simplified version of the code:

Client Side:

initiate () {
        return new Promise(resolve => {
            this.N_INCOMING = 0;
            this.N_OUTGOING = 0;
            this.rrCounter = 0;
            this.PULL_SOCKET.bind("tcp://*:" + this.MY_PORT, error => {
                utils.handleError(error);
                this.PUB_SOCKET.bind("tcp://*:" + (this.MY_PORT + 1), error => {
                    utils.handleError(error);
                    this.SUB_SOCKET.subscribe("");                    
                    this.SUB_SOCKET.connect(this.SERVER + ":" + (this.SERVER_PORT + 1),
                        error => {utils.handleError(error)});
                    this.PULL_SOCKET.on("message", (m) => this.handlePullSocket(m));
                    this.SUB_SOCKET.on("message", (m) => this.handleSubSocket(m));
                    this.SERVER_PUSH_SOCKET = zmq.socket("push");
                    this.SERVER_PUSH_SOCKET.connect(this.SERVER + ":" + this.SERVER_PORT,
                        error => {utils.handleError(error)});
                    this.sendHello();
                    resolve();
                });
            });
        });

Server side:

    initiate () {
        return new Promise(resolve => {
            this.PULL_SOCKET.bind(this.MY_IP + ":" + this.MY_PORT, err => {
                if (err) {
                    console.log(err);
                    process.exit(0);
                }
                this.PUB_SOCKET.bind(this.MY_IP + ":" + (this.MY_PORT + 1), err => {
                    if (err) {
                        console.log(err);
                        process.exit(0);
                    }
                    this.PULL_SOCKET.on("message", (m) => this.handlePullSocket(m));
                    resolve();
                });
            });
        });
    }

Client initiates the connection by sending the Hello Message. Server's listener function handlePullSocket should handle those messages.

Edit 2. 12/01/2021 As requested, I am adding the deployment/service configurations.

Client-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    kompose.cmd: kompose convert -f docker-compose-resolved.yml
    kompose.version: 1.19.0 (f63a961c)
  creationTimestamp: null
  labels:
    io.kompose.service: c1
  name: c1
  namespace: fishtrace
spec:
  replicas: 1
  selector:
    matchLabels:
      app: c1
  strategy:
    type: Recreate
  template:
    metadata:
      annotations:
        kompose.cmd: kompose convert -f docker-compose-resolved.yml
        kompose.version: 1.19.0 (f63a961c)
      creationTimestamp: null
      labels:
        app: c1
    spec:
      containers:
      - env:
        - name: NODO_ADDRESS
          value: 0.0.0.0
        - name: NODO_PUERTO
          value: "9999"
        - name: NODO_PUERTO_CADENA
          value: "8888"
        - name: SERVER_ADDRESS
          value: tcp://servidor
        - name: SERVER_PUERTO
          value: "7777"
        image: registrogeminis.com/docker_c1_rpi:latest
        name: c1
        ports:
          - containerPort: 8888
          - containerPort: 9999
        resources: {}
        volumeMounts:
        - mountPath: /app/vol
          name: c1-volume
        imagePullPolicy: Always
      restartPolicy: Always
      imagePullSecrets:
        - name: myregistrykey
      volumes:
        - name: c1-volume
          persistentVolumeClaim:
            claimName: c1-volume
status: {}

Client-service.yaml

apiVersion: v1
kind: Service
metadata:
  annotations:
    kompose.cmd: ./kompose convert
    kompose.version: 1.22.0 (955b78124)
  creationTimestamp: null
  labels:
    io.kompose.service: c1
  name: c1
spec:
  ports:
    - name: "9999"
      port: 9999
      targetPort: 9999
    - name: "8888"
      port: 8888
      targetPort: 8888
  selector:
    app: c1
  type: ClusterIP

Server-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    kompose.cmd: kompose convert -f docker-compose-resolved.yml
    kompose.version: 1.19.0 (f63a961c)
  creationTimestamp: null
  labels:
    io.kompose.service: servidor
  name: servidor
  namespace: fishtrace
spec:
  replicas: 1
  selector:
    matchLabels:
      app: servidor
  strategy:
    type: Recreate
  template:
    metadata:
      annotations:
        kompose.cmd: kompose convert -f docker-compose-resolved.yml
        kompose.version: 1.19.0 (f63a961c)
      creationTimestamp: null
      labels:
        app: servidor
    spec:
      containers:
      - env:
        - name: SERVER_ADDRESS
          value: tcp://*
        - name: SERVER_PUERTO
          value: "7777"
        image: registrogeminis.com/docker_servidor_rpi:latest
        name: servidor
        ports:
          - containerPort: 7777
          - containerPort: 7778
        resources: {}
        volumeMounts:
        - mountPath: /app/vol
          name: servidor-volume
        imagePullPolicy: Always
      restartPolicy: Always
      imagePullSecrets:
        - name: myregistrykey
      volumes:
      - name: servidor-volume
        persistentVolumeClaim:
          claimName: servidor-volume
status: {}

Server-service.yaml

apiVersion: v1
kind: Service
metadata:
  annotations:
    kompose.cmd: ./kompose convert
    kompose.version: 1.22.0 (955b78124)
  creationTimestamp: null
  labels:
    io.kompose.service: servidor
  name: servidor
spec:
  ports:
    - name: "7777"
      port: 7777
      targetPort: 7777
  selector:
    app: servidor
  type: ClusterIP
1 Answers

At the end it was a DNS problem. It was always a DNS problem. Thanks to @Matt for pointing out the problem.

In the official Kubernetes DNS doc they state that there is a known issue with systems that use /etc/resolv.conf as a link to the real configuration file, /run/systemd/resolve/resolv.conf in my case. It is a well-known problem, and the recommended solution is to update the kubelet's configuration to point to /run/systemd/resolve/resolv.conf.

To do so, I added the line resolvConf:/run/systemd/resolve/resolv.conf in /var/lib/kubelet/config.yaml. I also edited /etc/kubernetes/kubelet.conf just to be sure. Finally, you are supposed to reload the service executing sudo systemctl daemon-reload && sudo systemctl restart kubelet to propagate the changes.

However, I already did that before asking in SE. And it did not seem to work. I had to restart the whole cluster to make the changes take effect. Then the DNS worked perfectly and the ZMQ sockets behaved as expected.

Update 31/04/2021: I discovered that you have to forcefully restart the coredns kubernetes' service to actually propagate the changes. So in the end kubectl rollout restart coredns -n kube-system after restarting the kubelet service was enough.

Related