How to leverage the world-size parameter for DistributedDataParallel in Pytorch example for multiple GPUs?

user_4993513 0
Viewed 548

I am running this Pytorch example on a g2.2xlarge AWS machine. So, when I run time python imageNet.py ImageNet2, it runs well with the following timing:

real    3m16.253s
user    1m50.376s
sys 1m0.872s

However, when I add the world-size parameter, it gets stuck and does not execute anything. The command is as follows: time python imageNet.py --world-size 2 ImageNet2

So, how do I leverage the DistributedDataParallel functionality with the world-size parameter in this script. The world-size parameter is nothing but number of distributed processes.

Do I spin up another similar instance for this purpose? If yes, then how do the script recognize the instance? Do I need to add some parameters like the instance's IP or something?

pytorch amazon-ec2 gpu python
1 Answers

World size argument is the number of nodes in your distributed training, so if you set the world size to 2 you need to run the same command with a different rank on the other node. If you just want to increase the number of GPUs on a single node you need to change ngpus_per_node instead. Take a look at the multiple node example in this Readme.

user_52149980
Related