We have a kafka sdk written over apache-kafka (2.7.0) that we use to produce and consume messages to kafka topics.
By default the configuration is like this -
- Auto commit is set to false
- We use commitSync() for offsets
- poll frequency for consumers is 1000 ms
- max.poll.records is set to 2
- Consumers are single threaded and single consumer runs per instance/pod (we use EKS)
Now, there is a order service that produces order-created message to order topic and it is consumed by another service that fulfils the order fulfil service. The fulfilment logic takes on an average 20s to process this message (too high!).
Because of this even if we have 10 partitions in the topic and 10 application pods / consumers running (they all belong to same consumer group), we can only process 3 messages per minute per consumer (30 messages per minute overall).
The problem in rate of message production at peak is around 300 per minute. Even if we scale to 50 partitions with 50 consumers, we can only process 150 per minute. And even here, each consumer remains underutilized in terms of cpu and memory usage.
Because of this, over time, there is a huge build up in consumer lag.
How do we scale to solve this problem? We can't have 100s of underutilized consumers running as that is not cost effective. Please help with any pointers to solve this.
PS. : We are looking into how to optimize the consumer that is taking 20s on average, but it will take time and we need a short term solution for this that is cost effective as well.
