We have an EKS cluster running our API and an S3 bucket having thousands of objects ( at the moment ~3K) being updated weekly at the same time as part of another batch job. Every time there is an update, we want to POST the objects to the API to process the objects. The average processing time per object is ~5mins.
So far using S3 events we deliver to an SQS queue and we want to find an event driven solution to notify EKS API to poll from the queue when new messages occur. Using Lambda (with the queue configured as event source mapping) to POST the messages to the API doesn't seem to be as scalable as we thought, due to the (relatively) limited lambda concurrent executions (quota) and increasing number of objects (e.g., next week 3K may be 4 or 5K) that makes it hard to estimate.
What we want ideally is to notify the pods to poll from the queue in batches (say, of size 10) when new messages occur. Are there any designs or services to achieve this in AWS?