Dears,
Our applications are disconnecting from Kafka cluster. Applications logs show warning like this one:
14:15:24.445 [kafka-producer-network-thread | producer-1] WARN o.a.k.c.NetworkClient - [Producer clientId=producer-1] Bootstrap broker A.B.C.D:9093 (id: -2 rack: null) disconnected
On the side of Kafka cluster there is nothing in the logs.
log4j.properties looks like this one:
cat log4j.properties|grep -v "#"
log4j.rootLogger=INFO, kafkaAppender
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=[%d] %p %m (%c)%n
log4j.appender.kafkaAppender=org.apache.log4j.RollingFileAppender
log4j.appender.kafkaAppender.MaxFileSize=50MB
log4j.appender.kafkaAppender.MaxBackupIndex=4
log4j.appender.kafkaAppender.File=/var/log/kafka/kafka-server.log
log4j.appender.kafkaAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.kafkaAppender.layout.ConversionPattern=[%d] %p %m (%c)%n
log4j.appender.stateChangeAppender=org.apache.log4j.RollingFileAppender
log4j.appender.stateChangeAppender.MaxFileSize=50MB
log4j.appender.stateChangeAppender.MaxBackupIndex=4
log4j.appender.stateChangeAppender.File=/var/log/kafka/state-change.log
log4j.appender.stateChangeAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.stateChangeAppender.layout.ConversionPattern=[%d] %p %m (%c)%n
log4j.appender.controllerAppender=org.apache.log4j.RollingFileAppender
log4j.appender.controllerAppender.MaxFileSize=50MB
log4j.appender.controllerAppender.MaxBackupIndex=4
log4j.appender.controllerAppender.File=/var/log/kafka/controller.log
log4j.appender.controllerAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.controllerAppender.layout.ConversionPattern=[%d] %p %m (%c)%n
log4j.logger.kafka=WARN, kafkaAppender
log4j.appender.requestAppender=org.apache.log4j.RollingFileAppender
log4j.appender.requestAppender.MaxFileSize=50MB
log4j.appender.requestAppender.MaxBackupIndex=4
log4j.appender.requestAppender.File=/var/log/kafka/kafka-request.log
log4j.appender.requestAppender.layout=org.apache.log4j.PatternLayout
log4j.appender.requestAppender.layout.ConversionPattern=[%d] %p %m (%c)%n
log4j.logger.kafka.controller=DEBUG, controllerAppender
log4j.additivity.kafka.controller=false
log4j.logger.state.change.logger=DEBUG, stateChangeAppender
log4j.additivity.state.change.logger=false
server.properties file look like this one:
broker.id=2
delete.topic.enable=true
auto.create.topics.enable=false
listeners=INSIDE://:9091,OUTSIDE1://:9093,OUTSIDE2://:9092
advertised.listeners=INSIDE://kafka-nX:9091,OUTSIDE1://kafka-nX:9093,OUTSIDE2://kafka-nX:9092
listener.security.protocol.map=INSIDE:SASL_SSL,OUTSIDE1:SASL_SSL,OUTSIDE2:PLAINTEXT
inter.broker.listener.name=INSIDE
sasl.enabled.mechanisms=SCRAM-SHA-512
sasl.mechanism.inter.broker.protocol=SCRAM-SHA-512
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
allow.everyone.if.no.acl.found=true
ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
ssl.secure.random.implementation=SHA1PRNG
super.users=User:admin
ssl.client.auth=none
ssl.keystore.location=/etc/pki/CA/certs/kafka.jks
ssl.keystore.password=password
ssl.key.password=kafkakafka
ssl.truststore.location=/etc/pki/CA/certs/truststore.p12
ssl.truststore.password=password
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
offsets.topic.replication.factor=3
transaction.state.log.replication.factor=3
transaction.state.log.min.isr=3
log.dirs=/kafka_data
num.partitions=1
num.recovery.threads.per.data.dir=1
log.flush.interval.messages=10000
log.flush.interval.ms=1000
log.retention.hours=168
log.retention.bytes=1073741824
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=A.B.C.D:2181,A.B.C.D+1:2181,A.B.C.D+2:2181
zookeeper.connection.timeout.ms=6000
Kafka cluster version: 2.8.1 Kafka client version: 3.0.1
How to monitor/analyze such cases?
How to configure log4j.properties file on the Kafka cluster side to see such disconnected sessions?
Do you have any idea what can be wrong?
Best Regards, Dan