Currently we are storing some real time application traffic data into S3 files that are parsed by microservices (kubernetes's Pods) as soon it is stored. We receive around 300 files (approx > 2 GB size) in every minute. Currently our existing solution (with Horizontal podscaling) is not processing the data in our defined time limit.
As it is realtime traffic that need to be parsed ASAP (100 ms latency), so we are trying to explore if we have some other strategy to parse these real time traffic from S3 bucket quickly.