Kafka expectantly shutting down. License topic could not be created

user_470184 0
Viewed 3816

On Kafka startup multiple messages are logged to kafka/logs/kafkaServer.out and contain:

INFO [Admin Manager on Broker 0]: Error processing create topic request CreatableTopic(name='_confluent-license', numPartitions=1, replicationFactor=3, assignments=[], configs=[CreateableTopicConfig(name='cleanup.policy', value='compact'), CreateableTopicConfig(name='min.insync.replicas', value='2')]) (kafka.server.AdminManager)

After approx 15 minutes Kafka shuts down and outputted to kafka/logs/kafkaServer.out is :

org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 1.
[2020-12-08 04:04:15,951] ERROR [KafkaServer id=0] Fatal error during KafkaServer startup. Prepare to shutdown

(kafka.server.KafkaServer) org.apache.kafka.common.errors.TimeoutException: License topic could not be created Caused by: org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 1. [2020-12-08 04:04:15,952] INFO [KafkaServer id=0] shutting down (kafka.server.KafkaServer)

It appears Kafka shuts down because the replication factor is set to 3 for the topic _confluent-license ? I'm not creating the topic _confluent-license, is this created as part of Kafka startup for licensing check ?

In attempt to fix I've modified /v5.5.0/etc/kafka/server.properties so that replication factor is 1 for internal topics:

############################# Internal Topic Settings  #############################
# The replication factor for the group metadata internal topics "__consumer_offsets" and "__transaction_state"
# For anything other than development testing, a value greater than 1 is recommended for to ensure availability such as 3.
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1

instead of 3 :

#offsets.topic.replication.factor=3
#transaction.state.log.replication.factor=3

But this does not fix the issue and same logs are generated. The replication factor of __consumer_offsets is still 3. How to reduce the replication factor of topic _confluent-license from 3 to 1 ? Or could there be an alternative issue that is causing Kafka to shutdown ?

confluent-platform apache-kafka
3 Answers

You should change the property confluent.license.topic.replication.factor by default it is 3.

user_94964480

(kafka.server.KafkaServer) org.apache.kafka.common.errors.TimeoutException: License topic could not be created Caused by: org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 1. [2020-12-08 04:04:15,952] INFO [KafkaServer id=0] shutting down (kafka.server.KafkaServer)

The above error is due to the license topic having a default replication factor as 3. The same can be configured with confluent.license.topic.replication.factor to be equal to 1 if you have only 1 broker. The documentation for the same is here.

[2020-12-08 07:46:02,241] ERROR Error checking or creating metrics topic (io.confluent.metrics.reporter.ConfluentMetricsReporter) org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 2 larger than available brokers: 1.

The above error is due to the Confluent Metrics Reporter being enabled. The replication factor for the metric topic is defaulted at 3 and can be configured with confluent.metrics.reporter.topic.replicas to be 1 if you have just one broker. The documentation for the same is here.

user_98403180

The property confluent.license.topic.replication.factor is not working as expected.

Instead, we can try anyone of these properties confluent.license.replication.factor=1 or confluent.license.admin.replication.factor=1 if the number of broker is 1. This property is not in the documentation.

user_120954150
Related