Firehose to S3 Vs Direct PUT to S3

Viewed 458

I have a requirement to PUT about 20 records per second to an S3 bucket.

That comes to roughly 20 * 60 * 60 * 24 * 30 = 51,840,000 PUTs per month.

I do not need any transformations but I would certainly want the PUTs to be GZIPped and partitioned by year/date/month/hour.

Option 1 - Just do PUTObjects on S3

  1. Price comes to about ~$260 a month
  2. I would have to do the GZIP/Partitions etc on the client side

Option 2 - Introduce a Firehose and wire it to S3

  1. And let's say I buffer only once in 10 minutes then that is about 6 * 24 * 30 = 4,320 PUTs. Price of S3 comes down to $21. With each record about 20 KB Firehose pricing is about 1000GB * 0.029 comes to about $30. So total pricing is $51. Costs for data transfer / storage etc are same in both approaches I believe.
  2. Firehose provides GZip/Partitions/buffering for me OOTB

It appears like Option 2 is the best for my use case. Am I missing something here? Thanks for looking!

0 Answers
Related