Docker swarm network debugging

Question

I have created a service in docker using docker swarm and stack, but somehow my application running in a service replica is not able to connect to other replicas.

Here is more detail. I have three nodes: one manage and two workers. At the same node as manager I run a registry, which is the source for the image which I deploy to the workers using docker stack.

Here is how my configuration file for docker stack looks like:

version: "3.2"
services:
  compute:
    image: openmpi-docker.registry:443/openmpi
    ports:
      - "2222:22"
    deploy:
      replicas: 4
      restart_policy:
        condition: none
      placement:
        constraints: [node.role != manager]
    networks:
      - openmpi-network
networks:
  openmpi-network:
    driver: overlay
    attachable: true

As you can guess the image contains an openmpi distribution. It starts sshd and opens port 22.

I can connect to one of the the services using following command:

ssh -p 2222 -i ./user-ssh/docker_id

Then I try to run a simple test. I figure out what are the IP addresses of other replicas and try to connect them over ssh:

for i in $(seq 3 6) ; do ssh 10.0.1.$i hostname ; done

This works as expected I see 4 different hostnames.

So my next test involves mpi.

mpirun -H 10.0.1.3,10.0.1.4,10.0.1.5,10.0.1.6 hostname

This command is supposed to do exactly the same, with the exception that it should use mpirun instead of ssh. But somehow this fails.

So I want to see where exactly mpirun hangs using strace, but it is impossible to do so with SYS_PTRACE capability. And there is seemingly no way to set it if you use swarm. So I try to create another container, not in swarm, which shares the same overlay network:

docker run --network openmpi_openmpi-network   --rm -it openmpi-docker.registry:443/openmpi bash

Here openmpi_openmpi-network is the name of the network created by "docker stack". And originally I planned to run it with "--privileged", but it turned out that I do not need that.

Then I switch from the root to a normal user and run the very same mpi command with the very same IP addresses, and the command works. So I do not run in the very same problem.

In other cases the solution to similar problem is just to turn off the firewall, but I'm pretty sure I don't have it inside containers. I created the image myself from debian:9 image and the only server I installed is sshd. And still this wouldn't explain the difference between starting the container in swarm and starting the container using docker run.

This makes me wonder. First how can I debug such kind of problem? Second what is so different between starting the service in swarm and starting it manually in regard to networking?

I would be grateful if you could help me answering these two questions.

Update

I tried to run mpirun with "--mca oob_base_verbose 100" and got following result. The logs are relatively long, but the key difference is in behaviour of the node which initiates the communication.

I a working log at some mpirun of the initiator node prints following:

...
[3365816c2c1e:00033] [[39982,0],2] oob:tcp:init adding 10.0.1.4 to our list of V4 connections
[3365816c2c1e:00033] [[39982,0],2] TCP STARTUP
[3365816c2c1e:00033] [[39982,0],2] attempting to bind to IPv4 port 0
[3365816c2c1e:00033] [[39982,0],2] assigned IPv4 port 57161
[3365816c2c1e:00033] mca:oob:select: Adding component to end
[3365816c2c1e:00033] mca:oob:select: checking available component usock
[3365816c2c1e:00033] mca:oob:select: Querying component [usock]
[3365816c2c1e:00033] oob:usock: component_available called
[3365816c2c1e:00033] [[39982,0],2] USOCK STARTUP
[3365816c2c1e:00033] SUNPATH: /tmp/openmpi-sessions-1000@3365816c2c1e_0/39982/0/usock
[3365816c2c1e:00033] mca:oob:select: Inserting component
[3365816c2c1e:00033] mca:oob:select: Found 2 active transports

[3365816c2c1e:00033] [[39982,0],2]: set_addr to uri 2620260352.0;usock;tcp://10.0.1.7,172.18.0.5:48695
[3365816c2c1e:00033] [[39982,0],2]:set_addr checking if peer [[39982,0],0] is reachable via component usock
[3365816c2c1e:00033] [[39982,0],2]: peer [[39982,0],0] is NOT reachable via component usock
[3365816c2c1e:00033] [[39982,0],2]:set_addr checking if peer [[39982,0],0] is reachable via component tcp
[3365816c2c1e:00033] [[39982,0],2] oob:tcp: ignoring address usock
[3365816c2c1e:00033] [[39982,0],2] oob:tcp: working peer [[39982,0],0] address tcp://10.0.1.7,172.18.0.5:48695
[3365816c2c1e:00033] [[39982,0],2] PASSING ADDR 10.0.1.7 TO MODULE
[3365816c2c1e:00033] [[39982,0],2]:tcp set addr for peer [[39982,0],0]
[3365816c2c1e:00033] [[39982,0],2] PASSING ADDR 172.18.0.5 TO MODULE
[3365816c2c1e:00033] [[39982,0],2]:tcp set addr for peer [[39982,0],0]
...

But in non working case the initiator node stops at "Found 2 active transports" message and never prints anything else.

Update 2

I also figured out that hostname works if I add "--mca oob_tcp_if_include 10.0.1.0/24", so that the full command is:

mpirun -H 10.0.1.3,10.0.1.4,10.0.1.5,10.0.1.6  --mca oob_tcp_if_include 10.0.1.0/24  hostname

Still I do not really understand why and how to avoid the need of specifying the subnet.

Docker swarm network debugging

0 Answers