I'm doing some tests on my producer (springboot), sending requests every 30 seconds for 1 hour.
I notice that after a few minutes (~10 minutes), with TRACE enabled, I have the error below:
2022-07-18 19:15:25.025 DEBUG 12991 --- [streams-kafka-1] o.apache.kafka.common.network.Selector : [Producer clientId=event-streams-kafka-1] Connection with localhost/127.0.0.1 disconnected
java.io.EOFException: null
at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:97) ~[kafka-clients-3.0.0.jar:na]
at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:452) ~[kafka-clients-3.0.0.jar:na]
at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:402) ~[kafka-clients-3.0.0.jar:na]
at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:674) [kafka-clients-3.0.0.jar:na]
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:576) [kafka-clients-3.0.0.jar:na]
at org.apache.kafka.common.network.Selector.poll(Selector.java:481) [kafka-clients-3.0.0.jar:na]
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:551) [kafka-clients-3.0.0.jar:na]
at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:328) [kafka-clients-3.0.0.jar:na]
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:243) [kafka-clients-3.0.0.jar:na]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_311]
2022-07-18 19:15:25.027 TRACE 12991 --- [streams-kafka-1] o.apache.kafka.common.network.Selector : [Producer clientId=event-streams-kafka-1] Read from closing channel failed, ignoring exception
java.io.EOFException: null
at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:97) ~[kafka-clients-3.0.0.jar:na]
at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:452) ~[kafka-clients-3.0.0.jar:na]
at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:402) ~[kafka-clients-3.0.0.jar:na]
at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:674) [kafka-clients-3.0.0.jar:na]
at org.apache.kafka.common.network.Selector.maybeReadFromClosingChannel(Selector.java:700) [kafka-clients-3.0.0.jar:na]
at org.apache.kafka.common.network.Selector.close(Selector.java:935) [kafka-clients-3.0.0.jar:na]
at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:625) [kafka-clients-3.0.0.jar:na]
at org.apache.kafka.common.network.Selector.poll(Selector.java:481) [kafka-clients-3.0.0.jar:na]
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:551) [kafka-clients-3.0.0.jar:na]
at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:328) [kafka-clients-3.0.0.jar:na]
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:243) [kafka-clients-3.0.0.jar:na]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_311]
2022-07-18 19:15:25.027 DEBUG 12991 --- [streams-kafka-1] org.apache.kafka.clients.NetworkClient : [Producer clientId=event-streams-kafka-1] Node -2 disconnected.
I checked in Kafka documentations, that the connections.max.idle.ms parameter could solve my idle disconnects problem.
I changed the connections.max.idle.ms=-1 parameter on the application side and got no improvement.
When I made the change from the broker side, the error stopped occurring.
But my big question is, what makes Kafka understand that my connection is idle? Since my application has been sending requests to Kafka every 30 seconds, is there any way to send a signal to kafka to prevent it from closing my connection?
This is a problem for me, as changing this parameter on the broker side will affect all applications that use Kafka in a productive environment. I don't feel comfortable with that.