I have a requirement to PUT about 20 records per second to an S3 bucket.
That comes to roughly 20 * 60 * 60 * 24 * 30 = 51,840,000 PUTs per month.
I do not need any transformations but I would certainly want the PUTs to be GZIPped and partitioned by year/date/month/hour.
Option 1 - Just do PUTObjects on S3
- Price comes to about ~$260 a month
- I would have to do the GZIP/Partitions etc on the client side
Option 2 - Introduce a Firehose and wire it to S3
- And let's say I buffer only once in 10 minutes then that is about
6 * 24 * 30 = 4,320PUTs. Price of S3 comes down to $21. With each record about 20 KB Firehose pricing is about 1000GB * 0.029 comes to about $30. So total pricing is $51. Costs for data transfer / storage etc are same in both approaches I believe. - Firehose provides GZip/Partitions/buffering for me OOTB
It appears like Option 2 is the best for my use case. Am I missing something here? Thanks for looking!