Does SageMaker Object Detection support distributed training?

Viewed 18

I'd like to shorten the time of training jobs that use SageMaker's built-in Object Detection algorithm by using distributed training (instance count > 1). I tried to do so by splitting up my training data into multiple RecordIO files instead of just one. I then used ShardedByS3Key for the train channel. The training job failed with this error "Multiple RecordIO files found in the train channel. Only single RecordIO file is supported for now."

This leads me to believe that the algorithm does not support faster training times through distributed training. Am I correct? Or is there some other way to leverage multiple training instances to speed up the training job duration?

1 Answers

You should be able to run the object detection algorithm on multi-GPU and mult-machine settings for distributed training. Just increase the number of instance to >1. For Record-IO format data sharding across GPU will be taken care by SageMaker. Also make sure to use the supported instance types -

https://docs.aws.amazon.com/sagemaker/latest/dg/object-detection.html

Related