Increase/Reduce max.poll.records dynamically

Viewed 15

Using Apache Kafka client from Java, is there any way to change a Kafka consumer config once has been created?.

I want to increase/reduce max.poll.records depending of the number of consumers in the cluster.

The only way is destroy, and recreate the consumer?. If that's the case, is bad since create a new consumer it will take time, and also the warmup.

1 Answers

It's possible, but you would need to call consumer.close() then create a new instance with new config properties.

I'm not sure what you mean by "warmup/take time" since your JVM would already be "warm" and the network connection to bootstrap to Kafka and fetch shouldn't be "slow".

If you really want to control resource utilization in Kafka cluster, use throttling quotas at the server side, rather than force clients to modify properties at runtime.

Related