We are developing a system in which we have to collect 10 Million records every day. We need a database solution, NoSQL. data is simple logs. We are using AWS for now. I want to know the cheaper solution from both available techs. AmazonDB or MongoDB.
We have 30 Tables that are collecting these logs.
I would recommend using fluentd or fluentbit to collect and stream the log into Kinesis Firehose, and the Kinesis Firehose will deliver the logs into S3. Then you can use Glue for the data catalog and use Athena for data aggregation or query.
If you would like to do transformations before putting the log into S3, you can take the advantage to use the Kinesis Firehose Processor for data transformation or format conversation with Lambda.
Here are some relevant articles https://aws.amazon.com/blogs/compute/building-a-scalable-log-solution-aggregator-with-aws-fargate-fluentd-and-amazon-kinesis-data-firehose/ https://aws.amazon.com/blogs/opensource/centralized-container-logging-fluent-bit/
