How to process large dataset with AWS Batch Transform

user_1185254 0
Viewed 475

I have gigabytes of data for which I want to make predictions using AWS SageMaker Endpoint. I have two main issues:

  • data comes as Excel files and AWS Batch Transform needs it to be in JSON format to be able to process it. Reading Excel just to save it as JSON is redundant and it's a big IO slowdown
  • Endpoint can only be invoked over HTTP which means a few MB payload limit - chunking into such small pieces slows things down as well

How can I tackle these issues?

Pipe Mode could be a potential solution but from I read it is used for training only. Is it possible to use Pipe Mode for inference to speed things up?

bigdata amazon-sagemaker amazon-web-services pipeline
1 Answers

I would recommend to perform some preprocessing and ETL operations either using Glue/EMR and then use mini-batches to send data to Batch transform.

A blog regarding the same can be found here

Thanks, Raghu

user_64115480
Related