How to set backend to ‘gloo’ on windows in Pytorch

Mo Balut 0
Viewed 60

I am trying to use two gpus on my windows machine, but I keep getting

raise RuntimeError("Distributed package doesn't have NCCL " "built in") RuntimeError: Distributed package doesn't have NCCL built in

I am still new to pytorch and couldnt really find a way of setting the backend to ‘gloo’. Any way to set backend= 'gloo' to run two gpus on windows.

distributed pytorch-lightning pytorch
1 Answers
from torch import distributed as dist

Then in your init of the training logic:

dist.init_process_group("gloo", rank=rank, world_size=world_size)

Update:

You should use python multiprocess like this:

class Trainer:
    def __init__(self, rank, world_size):
        self.rank = rank
        self.world_size = world_size
        self.log('Initializing distributed')
        os.environ['MASTER_ADDR'] = self.args.distributed_addr
        os.environ['MASTER_PORT'] = self.args.distributed_port
        dist.init_process_group("gloo", rank=rank, world_size=world_size)

if __name__ == '__main__':
    world_size = torch.cuda.device_count()
    mp.spawn(
        Trainer,
        nprocs=world_size,
        args=(world_size,),
        join=True)
maloletnik0
Related