I have implemented my PyTorch model with DistributedDataParallel for multi-GPU training on the same Linux machine according to the official instructions. However, it output the following error during the setup stage:
ProcessRaisedException:
-- Process 0 terminated with the following error:
Traceback (most recent call last):
File "/data/ddmg/prism/.conda/envs/hou-purple/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 59, in _wrap
fn(i, *args)
File "/data/ddmg/purple/relative_prevalence_benchmark/method_dl.py", line 111, in train_relative_estimator
setup(rank, world_size)
File "/data/ddmg/purple/relative_prevalence_benchmark/method_dl.py", line 35, in setup
dist.init_process_group(backend='nccl', rank=rank, world_size=world_size, timeout=timedelta(seconds=30))
File "/data/ddmg/prism/.conda/envs/hou-purple/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py", line 561, in init_process_group
_store_based_barrier(rank, store, timeout)
File "/data/ddmg/prism/.conda/envs/hou-purple/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py", line 221, in _store_based_barrier
raise RuntimeError(
RuntimeError: Timed out initializing process group in store based barrier on rank: 0, for key: store_based_barrier_key:1 (world_size=7, worker_count=1, timeout=0:00:30)
I used NCLL as the backend here. I have tried other backends but they all resulted in the same error. I have also tried to use a longer timeout limit of up to 30 minutes, but it didn't help.