I have a single machine with two graphic cards. I'd like to use both for training my model. In an attempt to make my life easier, I'm using pytorch lightning.
When I run my script without gpus, everything works fine:
trainer = Trainer(gpus=None)
But I'd like to run it on multiple gpus. Confusingly, the manual, first states that for one machine with multiple gpus you should use the 'dp' accelerator:
trainer = Trainer(gpus=2, accelerator='dp')
This starts the training session, but gives me an BroadcastBackward error that I only get when I try to use the dp accelerator (details below).
However the manual goes on to say that
DP use is discouraged by PyTorch and Lightning. Use DDP which is more stable and at least 3x faster
Sounds great! So I do:
trainer = Trainer(gpus=2, accelerator='ddp')
But now when I run trainer.fit, the trainer starts a second process (proc = subprocess.Popen(command, env=env_copy, cwd=cwd) in ddp_accelerator.py.
On the one hand this is expected behaviour (one process per gpu), but on the other hand, it really literary reruns my main script (it's not just re-imported, so protecting with if __name__ == "__main__": doesn't help). My script is an optimisation scheme, with the following flow:
- generate hyperparameters (based on results of earlier runs)
- build, train, evaluate model: return performance measure
- repeat from 1
This means that the second process gets different hyperparameters than the first, and this obviously throws an error.
Running on 1 gpu works fine by the way (training completes without errors):
trainer = Trainer(gpus=1, accelerator='ddp')
How should I approach this problem? Stick with 'dp' and fix my BroadcastBackward error? Or stick with ddp, but how can I then fix the script duplication error?
dp error details:
In file X, Line X, in forward self.RMr.lerp_(Mr.squeeze(), exponential_average_factor) RuntimeError: Output 0 of BroadcastBackward is a view and is being modified inplace. This view is the output of a function that returns multiple views. Such functions do not allow the output views to be modified inplace. You should replace the inplace operation by an out-of-place one.
system details: ubuntu, python: 3.8, pytorch-lightning: 1.0.7, pytorch: 1.6.0