In my case, the training data is too large to save in one single computer or one computing node in a cluster (due to the limited disk space in each node), so it is split into several parts and each part is saved in one computing node. Suppose there are 3 computing nodes: A, B, and C. The folder for saving the part 1 in A is the /data/training_data/part1/, the folder for saving the part 2 in B is the /data/training_data/part2/, and the folder for saving the part 3 in C is the /data/training_data/part3/.
Then, how do I train a convolutional neural network using the torch.nn.parallel.DistributedDataParallel in this circumstance?
Could you please give some advice? Thanks a lot!