I have a usecase like this
- Filter logs from a cloudwatch log group that matches a pattern
- Transform and push them to s3 periodically as csv
- Push the s3 files to snowflake warehouse(this integration exists)
- Use them for analytics, to build aggregated reports.
The challenge is that the volume of logs is huge, like million records per day. What's the best option to perform this? What are the pros and cons?
- cloudwatch > firehose subscription filter > firehose > lambda
transform > s3 - cloudwatch > lambda subscription filter > lambda transform using custom code to write to s3
- Any other options to build reports within AWS (but not involving too much cost) and then expose publicly?