I am using python and mpi4py, and have encountered a scenario I do not understand. The below code is a minimal working example mwe.py.
import numpy as np
from mpi4py import MPI
import time
import itertools
N=8
comm = MPI.COMM_WORLD
rank = comm.Get_rank()
N_sub = comm.Get_size()-1
get_sub = itertools.cycle(range(1, N_slaves+1))
if rank == 0:
print("I am rank {}".format(rank))
data = []
for i in range(N):
nums = np.random.normal(size=6)
data.append(nums)
sleep_short = 0.0001
sleep_long = 0.1
sleep_dict = {}
for r in range(1, N_sub+1):
sleep_dict[str(r)] = 1
data_idx = 0
while len(data) > data_idx:
r = next(get_sub)
if comm.iprobe(r):
useless_info = comm.recv(source=r)
print(useless_info)
comm.send(data[data_idx], dest=r)
data_idx += 1
print("data_idx {}".format(data_idx))
sleep_dict[str(r)] = 1
else:
sleep_dict[str(r)] = 0
if all(value == 0 for value in sleep_dict.values()):
time.sleep(sleep_long)
else:
time.sleep(sleep_short)
for r in range(1, N_sub+1):
comm.send('Done', dest=r)
else:
print("rank {}".format(rank))
######################
# vvv This is the statement in question
######################
comm.Barrier()
while True:
comm.send("I am done", dest=0)
model = comm.recv(source=0)
if type(model).__name__ == 'str':
break
MPI.Finalize()
sys.exit(0)
When run with mpirun -np 4 python mwe.py, this code generates an array containing lists of random numbers, and then distributes these lists to the "sub" ranks until all arrays have been sent. Understandably, if I insert a comm.Barrier() call near the bottom (where I have indicated in the code), the code no longer completes execution, as the sub ranks (not equal to 0) never get to the statement where they are to receive what is being sent from rank 0. Rank 0 keeps trying to find a rank to pass the array to, but never does since the other ranks are held up, and the code hangs.
This makes sense to me. What doesn't make sense to me is that with the comm.Barrier() statement included, the preceding print statement also does not execute. Based on my understanding, the sub ranks should proceed normally until they hit the barrier statement, and then wait there until all of the ranks 'catch up', which in this case never happens because rank 0 is in its own loop. If this is the case, the preceding print statement should be executed, as it comes before those ranks have gotten to the barrier line. So why does the statement not get printed? Can anyone explain where my understanding fails?