DynamoDB vs ElasticSearch vs S3 - which service to use for superfast get/put 10-20MB files?

user_1986034 0
Viewed 3097

I have backend that recieves, stores and serves 10-20 MB json files. Which service should I use for superfast put and get (I cannot break the file in smaller chunks)? I dont have to run queries on these files just get them, store them and supply them instantly. The service should scale to tens of thousands of files easily. Ideally I should be able to put the file in 1-2 seconds and retrieve it in the same time.

I feel s3 is the best option and elastic search the second best option. Dyanmodb doesnt allow such object size. What should I use? Also, is there any other service? Mongodb is a possible solution but i dont see that on AWS, so something quick to setup would be great.

Thanks

amazon-dynamodb elasticsearch mongodb amazon-web-services amazon-s3
1 Answers

I don't think you should go for Dynamo or ES for this kind of operation.

After all, what you want is to store and serve it, not going into the file's content which both Dynamo and ES would waste time to do.

My suggestion is to use AWS Lambda + S3 to optimize for cost S3 does have some small downtime after putting till the file is available though ( It get bigger, minutes even, when you have millions of object in a bucket )

If downtime is important for your operation and total throughput at any given moment is not too huge, You can create a server ( preferably EC2) that serves as a temporary file stash. It will

  • Receive your file
  • Try to upload it to S3
  • If the file is requested before it's available on S3, serve the file on disk
  • If the file is successfully uploaded to S3, serve the S3 url, delete the file on disk
user_38443530
Related