I am running my code using MPI on a cluster. My code runs as a task in a docker running in swarm mode. Steps I follow to run my code:
- Create a overlay network
- Run docker in swarm mode
- Start a docker service (replicas = 4) using below command:
docker service create --name mpiser --network mpinet --replicas 4 mpitest:latest
- My test code is a simple python script having:
from mpi4py import MPI
import subprocess
import time
comm = MPI.COMM_WORLD
sizeComm = comm.Get_size()
rank = comm.Get_rank()
while True:
print("Rank:",rank,"Hostname:",subprocess.check_output(['hostname']))
time.sleep(2)
- I find the ip address of the tasks launched as part of the service
- exec into one of the containers
- create a "hosts" file with the ip addresses I found
- Launch the test code using below command:
mpirun --allow-run-as-root -n 33 --hostfile hosts --mca btl_tcp_if_exclude eth1,lo python3 /home/test.py
This works fine and I can see the prints from all the containers within the swarm.
However, if I expose one of the ports while creating service with below command
docker service create --name mpiser -p 3000:3000 --network mpinet --replicas 4 mpitest:latest
The mpirun command fails with below errors:
------------------------------------------------------------
A process or daemon was unable to complete a TCP connection
to another process:
Local host: 8d3c60280396
Remote host: cc2da25814cc
This is usually caused by a firewall on the remote host. Please
check that any firewall (e.g., iptables) has been disabled and
try again.
------------------------------------------------------------
I tried using --mca btl_tcp_if_include to include only the interface that shows the ip address I added in the hosts file I tried using --mca btl_tcp_if_exclude to exclued other interfaces that does not have the ip address I added in hosts file Both these did not help. Any suggestions on why exposing the port causes communication issue between the containers will be helpful