I believe that aws lambdas (serverless) are not good for batching, by definition they stop after 15 minutes of processing.
I have a Mongo Atlas (MongoDB Cloud Service) DB and I need to process a large dataset/collection several times a week, from a EC2 NodeJS app.
¿Which could be good architectural solutions for this to be efficient ?
Regards
MongoDB && Serverless architecture && batch processing
Viewed 79
1 Answers
Two key observations here. The size of the data and storage. MongoDB is partitioned and data can be read and processed in parallel.
For those both properties, Apache Spark is the best processing option. In AWS, couple of services provide this option and they are Amazon EMR and AWS Glue. From cost perspective and flexibility perspective, Amazon EMR is the best option.
What if you are not looking for parallel processing.? In that case, AWS Batch would be better option. Using AWS Batch you can run batch jobs with either EC2 or Fargate infra. You don't have to worry about provisioning and terminating the cluster with AWS Batch.
Related