Error: Some NCCL operations have failed or timed out

user_207661 0
Viewed 5283

While running a distributed training on 4 A6000 GPUs, I get the following error:

[E ProcessGroupNCCL.cpp:630] [Rank 3] Watchdog caught collective operation timeout: WorkNCCL(OpType=BROADCAST, Timeout(ms)=1800000) ran for 1803710 milliseconds before timing out.       
                                                                                                                                                        [E ProcessGroupNCCL.cpp:390] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. To avoid this inconsistency, we are taking the entire process down.                                                                                 

terminate called after throwing an instance of 'std::runtime_error'                                                                                                        
what():  [Rank 2] Watchdog caught collective operation timeout: 
WorkNCCL(OpType=BROADCAST, Timeout(ms)=1800000) ran for 1804406 milliseconds before timing out.        

[E ProcessGroupNCCL.cpp:390] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data. To avoid this inconsistency, we are taking the entire process down.

I use standard NVidia PyTorch docker. Interesting thing is that training works fine for small datasets but for the bigger datasets, I get this error. So I can confirm that the training code is correct and does work.

There is no actual runtime error or any other information to get an actual error messages anywhere.

nvidia-docker pytorch distributed gpu
3 Answers

Following two have solved the issue:

  • Increase default SHM (shared memory) for CUDA to 10g (I think 1g would have worked as well). You can do this in docker run command by passing --shm-size=10g. I also pass --ulimit memlock=-1.
  • export NCCL_P2P_LEVEL=NVL.

Debugging Tips

To check current SHM,

df -h
# see the row for shm

To see NCCL debug messages:

export NCCL_DEBUG=INFO

Run p2p bandwidth test for GPU to GPU communication link:

cd /usr/local/cuda/samples/1_Utilities/p2pBandwidthLatencyTest
sudo make
./p2pBandwidthLatencyTest

For A6000 4 GPU box this prints:

enter image description here

The matrix shows bandwith betweeb each pair of GPU and with P2P, it should be high.

user_2076610

For me, the problem turned out to be the torchrun command for PyTorch 1.10.1. I only needed to switch to the python -m torch.distributed.launch command and everything worked. I spent many hours on the StackOverflow and the PyTorch Forum but no one mentioned this solution, so I'm sharing it to save people time.

torchrun seems to be working fine for PyTorch 1.11 and above.

user_38535370
user_166584100
Related