Can I mount EFS in SageMaker processing job?

Stanislav Hordiyenko 0
Viewed 36

We are running some predictions using processing jobs in SageMaker. Before, we added some models into Docker image that was used by SageMaker processing job. Now, as we have more models (>= 6Gb), it is quite a bad design to add them to Docker image.

At first, we thought to download huge models from S3, but then realised, that it will cost a fortune. Then, an idea came to my mind. What if we use EFS with all heavy models and connect it to the running processing job so that python code can reference and use them?

I did a search and couldn't find any working solution for this. UI in AWS doesn't allow to connect EFS, and I couldn't find any working solution how to connect to EFS from the docker container. All suggest to use Docker volumes, but there is no way to configure this when create a processing job.

Question is is there any way to mount EFS to processing job in SageMaker? If no, what are the alternatives to host and use heavy models in the processing jobs? Maybe there are some other alternatives?

Thank you for any input.

amazon-sagemaker amazon-efs
1 Answers

So far I've not found a way to attach EFS - or any other shared storage volume - to a SM Processing Job (interestingly, you can do it for Training Jobs, though). If you find a way to do it, I will very like to hear about it :-)

At the same time, you have at least two alternatives here:

  1. Download models from S3 as you have mentioned (note: as long as you keep models and processing instances in the same AWS region, you will not pay too much for data transfer from/to S3!)

  2. Use SageMaker Model Registry (what - in extreme simplification - only adds some abstraction layer to S3 storage anyway).

If your model doesn't need GPU nor more memory than 10GB, you may also consider using Lambda's for inference (you can easily attach EFS there).

userMTM40
Related