Kafka creating more partitions than hardcoded in the properties file

user_7139447 0
Viewed 251

I have configured my producer as :

spring.cloud.stream.bindings.pc-abc-out-0.destination=pc-abc-my-topic
spring.cloud.stream.bindings.pc-abc-out-0.producer.partition-count=5
spring.cloud.stream.bindings.pc-abc-out-0.producer.header-mode=headers
spring.cloud.stream.bindings.pc-abc-out-0.producer.partition-count=10
spring.cloud.stream.bindings.pc-abc-out-0.producer.partitionKeyExpression=payload.key
spring.cloud.stream.kafka.bindings.pc-abc-out-0.producer.sync=true

However, in the kafka logs , I keep getting the error :


o.s.kafka.support.LoggingProducerListener - Exception thrown when sending a message with key='byte[14]' and payload='byte[253]' to *topic pc-abc-my-topic and partition 8*: org.apache.kafka.common.errors.TimeoutException: Topic pc-abc-my-topic not present in metadata after 60000 ms.

The highlight in the error message being : topic pc-abc-my-topic and partition 8

Why is it looking for partition 8 even though I have defined number of partitions as 5. Shouldn't the numbering be betweenn 0-4. I have several other error messages with partition number more than 5.

Earlier in my configuration I had added

spring.cloud.stream.kafka.binder.auto-add-partitions=true

But I removed it and we scaled down and scaled up the services. The issue still exists. Is this the case of stale config?

spring-cloud-config spring-kafka spring-cloud-stream-binder-kafka apache-kafka
2 Answers

The issue turned out to be this configuration:

spring.cloud.stream.bindings.pc-abc-out-0.producer.partitionKeyExpression=payload.key
spring.cloud.stream.kafka.bindings.pc-abc-out-0.producer.sync=true

The partition number being calculated because of payload.key is somwhow greater than the partitionCount value we configured and created on our infrastructure. Removing these two configuration stopped the issue.

user_71394470

i guess you create topic with 8 partition first. if topic already exist spring-kafka doesn't create new one with your partition configs.

Here doc https://docs.spring.io/spring-cloud-stream/docs/Brooklyn.RELEASE/reference/html/_apache_kafka_binder.html#:~:text=Default%3A%20Empty%20map.-,The,-Kafka%20binder%20will

The Kafka binder will use the partitionCount setting of the producer as a hint to create a topic with the given partition count (in conjunction with the minPartitionCount, the maximum of the two being the value being used). Exercise caution when configuring both minPartitionCount for a binder and partitionCount for an application, as the larger value will be used. If a topic already exists with a smaller partition count and autoAddPartitions is disabled (the default), then the binder will fail to start.
If a topic already exists with a smaller partition count and autoAddPartitions is enabled, new partitions will be added. If a topic already exists with a larger number of partitions than the maximum of (minPartitionCount and partitionCount), the existing partition count will be used.

Your real problem is Topic pc-abc-my-topic not present in metadata after 60000 ms. don't care about partition count.

Here solution for this

org.apache.kafka.common.errors.TimeoutException: Topic not present in metadata after 60000 ms

https://developpaper.com/topic-xxx-not-present-in-metadata-after-60000-ms/

user_61710250
Related