Kafka producer infinite log message: Error while fetching metadata with correlation id 18345 : {input-topic=UNKNOWN_TOPIC_OR_PARTITION}

user_4166822 0
Viewed 370

Upon trying to push message into unexisting kafka topic, kafka client keeps producing below log message:

Error while fetching metadata with correlation id 13599 : {input topic=UNKNOWN_TOPIC_OR_PARTITION}

I tried reducing retries setting to 5, but still, it keeps indefinitely printing this log message. Eventually then it crashes.

Is there a way to tell kafka to abort trying to get the topic metadata if topic not existing?

Here's my an excerpt of my producer config:

ProducerConfig values:  acks = 1    connections.max.idle.ms = 540000    delivery.timeout.ms = 120000    enable.idempotence = false      linger.ms = 0   max.block.ms = 9223372036854775807  max.in.flight.requests.per.connection = 1   max.request.size = 1048576  metadata.max.age.ms = 300000    metadata.max.idle.ms = 300000   reconnect.backoff.max.ms = 1000 reconnect.backoff.ms = 50   request.timeout.ms = 30000  retries = 5 retry.backoff.ms = 100  transaction.timeout.ms = 60000
kafka-producer-api apache-kafka
1 Answers

You cant really tell Kafka to "abort" in such a situation (as far as my online search shows), but you can alter the setting so the log is quite a bit less spammy.

Following two properties help minimizing those logs.

  1. retry.backoff.ms - which defaults to 100ms, is the interval in which sends are (re)-tried. Setting this to at least 1000 seems reasonable (for me)

  2. (ONLY FOR CONSUMERS) Since consumers can have this spammy logmessage too, there is the following property to alter this problem: max.poll.interval.ms - which defaults to 300s (5 minutes), is the time it keeps trying. After those 5 minutes, polls are no longer tried. Setting this to, like, 1 minute, seems like a reasonable value.

user_38820250
Related