I'm working on a deep learning project with about 700GB of table-like time series data in thousands of .csv files (each about 15MB).
All the data is on S3 and it needs some preprocessing before being fed into the model. The question is how to best go about automating the process of loading, preprocessing and training.
Is a custom keras generator with some built in preprocessing the best solution?
Efficient management of large amounts of data with SageMaker for training a keras model
Viewed 423
1 Answers
Preprocessing implies that this is something you might want to decouple from the model execution and run separately, possibly on a schedule or in response to new data flowing in.
If so, you'll probably want to do the preprocessing outside of SageMaker. You could orchestrate it using Glue, or you could write a custom job and run it through AWS Batch or alternatively on an EMR cluster.
That way, your Keras notebook can load the already preprocessed data, train and test through SageMaker.
With a little care, you should be able to perform at least some of the heavy lifting incrementally in the preprocessing step, saving both time and cost downstream in the Deep Learning pipeline.
Related