GPU 0 utilization higher than other GPUs on Amazon SageMaker SMDP (distributed Training)

Viewed 12

When using SageMaker Data Parallelism (SMDP), my team sees a higher utilization on GPU 0 compared to other GPUs. What can be the likely cause here? Does it have anything to do with the data loader workers that run on CPU? I would expect SMDP to shard the datasets equally.

0 Answers
Related