What are the best practices for distributed training with PyTorch custom containers (BYOC) in Amazon Sagemaker? I understand that PyTorch framework supports native distributed training or using the Horovod library for PyTorch.
Best Practices for Distributed Training with PyTorch custom containers (BYOC) in SageMaker
juvchan
0
Viewed 19
1 Answers
The recommended approach on Amazon SageMaker is to use the SageMaker built in Data Parallel and Model Parallel Libraries. When you use the Pytorch Deep Learning container provided by SageMaker, the library is built in and you can follow the below examples to get started with examples.
https://docs.aws.amazon.com/sagemaker/latest/dg/distributed-training-notebook-examples.html
If you are bringing your own container, follow the below link to add SageMaker Distributed training support to your container
Apart from this SageMaker also natively supports Pytorch DDP within the native Deep Learning Container used in Pytorch Estimator.
https://sagemaker.readthedocs.io/en/stable/frameworks/pytorch/sagemaker.pytorch.html
Related