I'm trying to figure out how to get callbacks to work well with pytorch-lightning and horovod on a 4 GPU machine. Currently, I'm just passing the callbacks into the Trainer and don't have any if hvd.rank() == 0 checks anywhere in the code. This is leading to errors, for example with wandb logging 4 separate times.
What's the proper way to use callbacks in horovod+pytorch-lightning so that we don't get these clashes? Do I need to check the rank in each of the callbacks before logging or saving a checkpoint? What about for LR scheduling? Should that only be performed on a single node or on all of them?