Kafka consumer that we have written using spring-kafka library and with @Listener annotation seems to be randomly skipping messages. This is not happening every time but enough number of times to make us investigate. Seems to be happening more around the time when events are published after some time gap i.e the consumer does not have any message to process for sometime.
We are using AWS MSK (AWS Managed kafka).
- Topic has replication factor of 2
- Ack mode of producer is all
- auto offset reset is 'earliest'
We have simultaneously tried using the kafka console consumer and that seems to be picking up messages.
Has anyone faced such an issue before ? What are some of the configurations we can tweak ?