I have a problem with zookeeper deployed on kubernetes using Confluentinc Docker Images.
Cluster setup:
3 nodes in digital ocean ( 4 CPU, 8 GB or RAM) as kafka dedicated nodepool. ( Only zookeeper and kafka are running here, and some monitoring daemon set).
If a zookeeper node/pod ( any node) restarts several times the zookeeper cluster is not accessible anymore, and to make it work I have to redeploy all the nodes.
Once the node rejoined the cluster, all zookeeper nodes start to show this kind of error: Each node has one kafka pod and one zookeeper pod.
Errors one each node: zookeeper-node-0 logs: [2020-10-29 00:29:20,852] INFO Refusing session request for client /10.244.4.86:47054 as it has seen zxid 0xd00000000 our last zxid is 0xc00000004 client must try another server (org.apache.zookeeper.server.ZooKeeperServer)
zookeeper-node-1 logs: 244.3.185:36040 as it has seen zxid 0xd00000000 our last zxid is 0xc00000004 client must try another server (org.apache.zookeeper.server.ZooKeeperServer).
zookeeper-node-2 logs: [2020-10-29 00:28:59,554] WARN Unable to read additional data from client sessionid 0x102508ff4380002, likely client has closed socket (org.apache.zookeeper.server.NIOServerCnxn)
although the zookeeper cluster seems to be up, I'm not able to get anything from it. If I try to use cli to list a path I'm getting a timeout.
Any Ideas?
Node 0 Full Logs:
[2020-10-29 00:25:16,976] INFO Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /var/lib/zookeeper/log/version-2 snapdir /var/lib/zookeeper/data/version-2 (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:25:16,977] INFO FOLLOWING - LEADER ELECTION TOOK - 33 MS (org.apache.zookeeper.server.quorum.Learner)
[2020-10-29 00:25:17,048] INFO Getting a diff from the leader 0xc00000004 (org.apache.zookeeper.server.quorum.Learner)
[2020-10-29 00:25:17,056] INFO Learner received NEWLEADER message (org.apache.zookeeper.server.quorum.Learner)
[2020-10-29 00:25:17,072] INFO Learner received UPTODATE message (org.apache.zookeeper.server.quorum.Learner)
[2020-10-29 00:25:17,080] INFO Configuring CommitProcessor with 1 worker threads. (org.apache.zookeeper.server.quorum.CommitProcessor)
[2020-10-29 00:25:45,993] WARN Got zxid 0xd00000001 expected 0x1 (org.apache.zookeeper.server.quorum.Learner)
[2020-10-29 00:25:45,993] INFO Creating new log file: log.d00000001 (org.apache.zookeeper.server.persistence.FileTxnLog)
[2020-10-29 00:25:55,308] INFO Refusing session request for client /10.244.4.177:45756 as it has seen zxid 0xd00000000 our last zxid is 0xc00000004 client must try another server (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:25:56,732] INFO Refusing session request for client /10.244.4.177:45758 as it has seen zxid 0xd00000000 our last zxid is 0xc00000004 client must try another server (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:25:58,591] INFO Refusing session request for client /10.244.4.177:45760 as it has seen zxid 0xd00000000 our last zxid is 0xc00000004 client must try another server (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:25:58,934] INFO Refusing session request for client /10.244.4.86:58236 as it has seen zxid 0xd00000000 our last zxid is 0xc00000004 client must try another server (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:26:00,280] INFO Refusing session request for client /10.244.4.177:45762 as it has seen zxid 0xd00000000 our last zxid is 0xc00000004 client must try another server (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:26:27,489] INFO Refusing session request for client /10.244.3.185:35932 as it has seen zxid 0xd00000000 our last zxid is 0xc00000004 client must try another server (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:26:28,694] INFO Refusing session request for client /10.244.3.185:35934 as it has seen zxid 0xd00000000 our last zxid is 0xc00000004 client must try another server (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:26:29,368] INFO Refusing session request for client /10.244.4.86:58254 as it has seen zxid 0xd00000000 our last zxid is 0xc00000004 client must try another server (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:26:42,629] INFO Refusing session request for client /10.244.4.177:45808 as it has seen zxid 0xd00000000 our last zxid is 0xc00000004 client must try another server (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:26:44,040] INFO Refusing session request for client /10.244.4.177:45814 as it has seen zxid 0xd00000000 our last zxid is 0xc00000004 client must try another server (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:26:45,968] INFO Refusing session request for client /10.244.3.185:35950 as it has seen zxid 0xd00000000 our last zxid is 0xc00000004 client must try another server (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:26:47,727] INFO Refusing session request for client /10.244.4.86:58274 as it has seen zxid 0xd00000000 our last zxid is 0xc00000004 client must try another server (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:27:37,538] INFO Refusing session request for client /10.244.4.86:58326 as it has seen zxid 0xd00000000 our last zxid is 0xc00000004 client must try another server (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:27:42,105] INFO Refusing session request for client /10.244.4.177:45854 as it has seen zxid 0xd00000000 our last zxid is 0xc00000004 client must try another server (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:27:46,458] INFO Refusing session request for client /10.244.3.185:36024 as it has seen zxid 0xd00000000 our last zxid is 0xc00000004 client must try another server (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:27:58,125] INFO Refusing session request for client /10.244.4.177:45870 as it has seen zxid 0xd00000000 our last zxid is 0xc00000004 client must try another server (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:28:01,240] INFO Refusing session request for client /10.244.3.185:36036 as it has seen zxid 0xd00000000 our last zxid is 0xc00000004 client must try another server (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:28:04,306] INFO Refusing session request for client /10.244.3.185:36040 as it has seen zxid 0xd00000000 our last zxid is 0xc00000004 client must try another server (org.apache.zookeeper.server.ZooKeeperServer)
Node 1 FULL LOGS:
[2020-10-29 00:25:45,993] WARN Got zxid 0xd00000001 expected 0x1 (org.apache.zookeeper.server.quorum.Learner)
[2020-10-29 00:25:45,994] INFO Creating new log file: log.d00000001 (org.apache.zookeeper.server.persistence.FileTxnLog)
[2020-10-29 00:26:14,664] INFO Refusing session request for client /10.244.4.86:46890 as it has seen zxid 0xd00000000 our last zxid is 0xc00000004 client must try another server (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:26:26,244] INFO Refusing session request for client /10.244.3.185:43540 as it has seen zxid 0xd00000000 our last zxid is 0xc00000004 client must try another server (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:26:29,979] INFO Refusing session request for client /10.244.3.185:43546 as it has seen zxid 0xd00000000 our last zxid is 0xc00000004 client must try another server (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:26:31,278] INFO Refusing session request for client /10.244.4.86:46902 as it has seen zxid 0xd00000000 our last zxid is 0xc00000004 client must try another server (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:26:32,518] INFO Refusing session request for client /10.244.4.86:46904 as it has seen zxid 0xd00000000 our last zxid is 0xc00000004 client must try another server (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:26:44,651] INFO Refusing session request for client /10.244.3.185:43558 as it has seen zxid 0xd00000000 our last zxid is 0xc00000004 client must try another server (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:26:47,391] INFO Refusing session request for client /10.244.3.185:43562 as it has seen zxid 0xd00000000 our last zxid is 0xc00000004 client must try another server (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:26:48,682] INFO Refusing session request for client /10.244.3.185:43564 as it has seen zxid 0xd00000000 our last zxid is 0xc00000004 client must try another server (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:26:49,204] INFO Refusing session request for client /10.244.4.86:46924 as it has seen zxid 0xd00000000 our last zxid is 0xc00000004 client must try another server (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:26:50,330] INFO Refusing session request for client /10.244.4.86:46928 as it has seen zxid 0xd00000000 our last zxid is 0xc00000004 client must try another server (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:27:12,233] INFO Refusing session request for client /10.244.4.177:40530 as it has seen zxid 0xd00000000 our last zxid is 0xc00000004 client must try another server (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:27:20,123] INFO Refusing session request for client /10.244.4.86:46946 as it has seen zxid 0xd00000000 our last zxid is 0xc00000004 client must try another server (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:27:21,836] INFO Refusing session request for client /10.244.4.86:46948 as it has seen zxid 0xd00000000 our last zxid is 0xc00000004 client must try another server (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:27:43,517] INFO Refusing session request for client /10.244.4.177:40560 as it has seen zxid 0xd00000000 our last zxid is 0xc00000004 client must try another server (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:27:45,270] INFO Refusing session request for client /10.244.3.185:43632 as it has seen zxid 0xd00000000 our last zxid is 0xc00000004 client must try another server (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:28:02,862] INFO Refusing session request for client /10.244.3.185:43648 as it has seen zxid 0xd00000000 our last zxid is 0xc00000004 client must try another server (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:28:22,019] INFO Refusing session request for client /10.244.3.185:43672 as it has seen zxid 0xd00000000 our last zxid is 0xc00000004 client must try another server (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:28:23,603] INFO Refusing session request for client /10.244.3.185:43674 as it has seen zxid 0xd00000000 our last zxid is 0xc00000004 client must try another server (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:28:43,276] INFO Refusing session request for client /10.244.3.185:43706 as it has seen zxid 0xd00000000 our last zxid is 0xc00000004 client must try another server (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:29:04,320] INFO Refusing session request for client /10.244.4.86:47030 as it has seen zxid 0xd00000000 our last zxid is 0xc00000004 client must try another server (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:29:11,616] INFO Refusing session request for client /10.244.4.86:47040 as it has seen zxid 0xd00000000 our last zxid is 0xc00000004 client must try another server (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:29:13,125] INFO Refusing session request for client /10.244.4.86:47042 as it has seen zxid 0xd00000000 our last zxid is 0xc00000004 client must try another server (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:29:14,923] INFO Refusing session request for client /10.244.4.86:47044 as it has seen zxid 0xd00000000 our last zxid is 0xc00000004 client must try another server (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:29:16,594] INFO Refusing session request for client /10.244.4.86:47046 as it has seen zxid 0xd00000000 our last zxid is 0xc00000004 client must try another server (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:29:19,303] INFO Refusing session request for client /10.244.4.86:47050 as it has seen zxid 0xd00000000 our last zxid is 0xc00000004 client must try another server (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:29:20,852] INFO Refusing session request for client /10.244.4.86:47054 as it has seen zxid 0xd00000000 our last zxid is 0xc00000004 client must try another server (org.apache.zookeeper.server.ZooKeeperServer)
Node 2 FULL LOGS:
[2020-10-29 00:22:38,980] INFO Synchronizing with Follower sid: 2 maxCommittedLog=0xc00000004 minCommittedLog=0xb00000001 lastProcessedZxid=0xd00000000 peerLastZxid=0xc00000004 (org.apache.zookeeper.server.quorum.LearnerHandler)
[2020-10-29 00:22:38,981] INFO Using committedLog for peer sid: 2 (org.apache.zookeeper.server.quorum.LearnerHandler)
[2020-10-29 00:22:38,981] INFO Sending DIFF zxid=0xc00000004 for peer sid: 2 (org.apache.zookeeper.server.quorum.LearnerHandler)
[2020-10-29 00:23:02,911] WARN Connection broken for id 2, my id = 3, error = (org.apache.zookeeper.server.quorum.QuorumCnxManager)
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1226)
[2020-10-29 00:23:02,911] ERROR Unexpected exception causing shutdown while sock still open (org.apache.zookeeper.server.quorum.LearnerHandler)
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:84)
at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:86)
at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:118)
at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:539)
[2020-10-29 00:23:02,912] WARN Interrupting SendWorker thread from RecvWorker. sid: 2. myId: 3 (org.apache.zookeeper.server.quorum.QuorumCnxManager)
[2020-10-29 00:23:02,913] WARN ******* GOODBYE /10.244.4.236:33882 ******** (org.apache.zookeeper.server.quorum.LearnerHandler)
[2020-10-29 00:23:02,915] WARN Interrupted while waiting for message on queue (org.apache.zookeeper.server.quorum.QuorumCnxManager)
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1308)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:85)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1143)
[2020-10-29 00:23:02,922] WARN Send worker leaving thread id 2 my id = 3 (org.apache.zookeeper.server.quorum.QuorumCnxManager)
[2020-10-29 00:23:19,562] INFO Received connection request from /10.244.4.195:33082 (org.apache.zookeeper.server.quorum.QuorumCnxManager)
[2020-10-29 00:23:19,576] INFO Notification: 2 (message format version), 2 (n.leader), 0xc00000004 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0xd (n.peerEPoch), LEADING (my state)0 (n.config version) (org.apache.zookeeper.server.quorum.FastLeaderElection)
[2020-10-29 00:23:19,653] INFO Follower sid: 2 : info : cp-kafka-cp-zookeeper-1.cp-kafka-cp-zookeeper-headless.default:2888:3888:participant (org.apache.zookeeper.server.quorum.LearnerHandler)
[2020-10-29 00:23:19,654] INFO On disk txn sync enabled with snapshotSizeFactor 0.33 (org.apache.zookeeper.server.ZKDatabase)
[2020-10-29 00:23:19,654] INFO Synchronizing with Follower sid: 2 maxCommittedLog=0xc00000004 minCommittedLog=0xb00000001 lastProcessedZxid=0xd00000000 peerLastZxid=0xc00000004 (org.apache.zookeeper.server.quorum.LearnerHandler)
[2020-10-29 00:23:19,654] INFO Using committedLog for peer sid: 2 (org.apache.zookeeper.server.quorum.LearnerHandler)
[2020-10-29 00:23:19,654] INFO Sending DIFF zxid=0xc00000004 for peer sid: 2 (org.apache.zookeeper.server.quorum.LearnerHandler)
[2020-10-29 00:24:52,134] WARN Connection broken for id 1, my id = 3, error = (org.apache.zookeeper.server.quorum.QuorumCnxManager)
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:1226)
[2020-10-29 00:24:52,134] ERROR Unexpected exception causing shutdown while sock still open (org.apache.zookeeper.server.quorum.LearnerHandler)
java.io.EOFException
at java.io.DataInputStream.readInt(DataInputStream.java:392)
at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:84)
at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:86)
at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:118)
at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:539)
[2020-10-29 00:24:52,135] WARN ******* GOODBYE /10.244.4.5:35810 ******** (org.apache.zookeeper.server.quorum.LearnerHandler)
[2020-10-29 00:24:52,134] WARN Interrupting SendWorker thread from RecvWorker. sid: 1. myId: 3 (org.apache.zookeeper.server.quorum.QuorumCnxManager)
[2020-10-29 00:24:52,137] WARN Interrupted while waiting for message on queue (org.apache.zookeeper.server.quorum.QuorumCnxManager)
java.lang.InterruptedException
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:1308)
at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$700(QuorumCnxManager.java:85)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:1143)
[2020-10-29 00:24:52,138] WARN Send worker leaving thread id 1 my id = 3 (org.apache.zookeeper.server.quorum.QuorumCnxManager)
[2020-10-29 00:25:16,868] INFO Received connection request from /10.244.4.117:53160 (org.apache.zookeeper.server.quorum.QuorumCnxManager)
[2020-10-29 00:25:16,941] INFO Notification: 2 (message format version), 1 (n.leader), 0xc00000004 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0xd (n.peerEPoch), LEADING (my state)0 (n.config version) (org.apache.zookeeper.server.quorum.FastLeaderElection)
[2020-10-29 00:25:16,942] INFO Notification: 2 (message format version), 2 (n.leader), 0xc00000004 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0xd (n.peerEPoch), LEADING (my state)0 (n.config version) (org.apache.zookeeper.server.quorum.FastLeaderElection)
[2020-10-29 00:25:16,943] INFO Notification: 2 (message format version), 1 (n.leader), 0xc00000004 (n.zxid), 0x2 (n.round), LOOKING (n.state), 1 (n.sid), 0xd (n.peerEPoch), LEADING (my state)0 (n.config version) (org.apache.zookeeper.server.quorum.FastLeaderElection)
[2020-10-29 00:25:17,033] INFO Follower sid: 1 : info : cp-kafka-cp-zookeeper-0.cp-kafka-cp-zookeeper-headless.default:2888:3888:participant (org.apache.zookeeper.server.quorum.LearnerHandler)
[2020-10-29 00:25:17,035] INFO On disk txn sync enabled with snapshotSizeFactor 0.33 (org.apache.zookeeper.server.ZKDatabase)
[2020-10-29 00:25:17,035] INFO Synchronizing with Follower sid: 1 maxCommittedLog=0xc00000004 minCommittedLog=0xb00000001 lastProcessedZxid=0xd00000000 peerLastZxid=0xc00000004 (org.apache.zookeeper.server.quorum.LearnerHandler)
[2020-10-29 00:25:17,035] INFO Using committedLog for peer sid: 1 (org.apache.zookeeper.server.quorum.LearnerHandler)
[2020-10-29 00:25:17,035] INFO Sending DIFF zxid=0xc00000004 for peer sid: 1 (org.apache.zookeeper.server.quorum.LearnerHandler)
[2020-10-29 00:25:41,812] WARN Unable to read additional data from client sessionid 0x10250d24d9f0001, likely client has closed socket (org.apache.zookeeper.server.NIOServerCnxn)
[2020-10-29 00:25:53,700] WARN Unable to read additional data from client sessionid 0x2025090352a0002, likely client has closed socket (org.apache.zookeeper.server.NIOServerCnxn)
[2020-10-29 00:25:57,706] WARN Unable to read additional data from client sessionid 0x102508ff4380002, likely client has closed socket (org.apache.zookeeper.server.NIOServerCnxn)
[2020-10-29 00:25:57,843] WARN Unable to read additional data from client sessionid 0x2025090352a0000, likely client has closed socket (org.apache.zookeeper.server.NIOServerCnxn)
[2020-10-29 00:26:08,374] INFO Expiring session 0x10250d24d9f0001, timeout of 30000ms exceeded (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:26:08,374] INFO Submitting global closeSession request for session 0x10250d24d9f0001 (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:26:11,410] WARN Unable to read additional data from client sessionid 0x2025090352a0000, likely client has closed socket (org.apache.zookeeper.server.NIOServerCnxn)
[2020-10-29 00:26:12,670] WARN Unable to read additional data from client sessionid 0x102508ff4380002, likely client has closed socket (org.apache.zookeeper.server.NIOServerCnxn)
[2020-10-29 00:26:13,530] WARN Unable to read additional data from client sessionid 0x2025090352a0002, likely client has closed socket (org.apache.zookeeper.server.NIOServerCnxn)
[2020-10-29 00:26:18,375] INFO Expiring session 0x10250d8e9830000, timeout of 30000ms exceeded (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:26:18,375] INFO Submitting global closeSession request for session 0x10250d8e9830000 (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:26:24,786] WARN Unable to read additional data from client sessionid 0x2025090352a0000, likely client has closed socket (org.apache.zookeeper.server.NIOServerCnxn)
[2020-10-29 00:26:27,047] WARN Unable to read additional data from client sessionid 0x2025090352a0002, likely client has closed socket (org.apache.zookeeper.server.NIOServerCnxn)
[2020-10-29 00:26:27,959] WARN Unable to read additional data from client sessionid 0x102508ff4380002, likely client has closed socket (org.apache.zookeeper.server.NIOServerCnxn)
[2020-10-29 00:26:40,736] WARN Unable to read additional data from client sessionid 0x2025090352a0002, likely client has closed socket (org.apache.zookeeper.server.NIOServerCnxn)
[2020-10-29 00:26:43,454] WARN Unable to read additional data from client sessionid 0x2025090352a0000, likely client has closed socket (org.apache.zookeeper.server.NIOServerCnxn)
[2020-10-29 00:26:45,990] WARN Unable to read additional data from client sessionid 0x102508ff4380002, likely client has closed socket (org.apache.zookeeper.server.NIOServerCnxn)
[2020-10-29 00:26:48,375] INFO Expiring session 0x30250d64f6e0000, timeout of 30000ms exceeded (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:26:48,375] INFO Submitting global closeSession request for session 0x30250d64f6e0000 (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:26:57,536] WARN Unable to read additional data from client sessionid 0x2025090352a0002, likely client has closed socket (org.apache.zookeeper.server.NIOServerCnxn)
[2020-10-29 00:27:02,514] WARN Unable to read additional data from client sessionid 0x2025090352a0000, likely client has closed socket (org.apache.zookeeper.server.NIOServerCnxn)
[2020-10-29 00:27:04,262] WARN Unable to read additional data from client sessionid 0x102508ff4380002, likely client has closed socket (org.apache.zookeeper.server.NIOServerCnxn)
[2020-10-29 00:27:10,996] WARN Unable to read additional data from client sessionid 0x2025090352a0002, likely client has closed socket (org.apache.zookeeper.server.NIOServerCnxn)
[2020-10-29 00:27:16,050] WARN Unable to read additional data from client sessionid 0x2025090352a0000, likely client has closed socket (org.apache.zookeeper.server.NIOServerCnxn)
[2020-10-29 00:27:18,028] WARN Unable to read additional data from client sessionid 0x102508ff4380002, likely client has closed socket (org.apache.zookeeper.server.NIOServerCnxn)
[2020-10-29 00:27:20,374] INFO Expiring session 0x30250d64f6e0001, timeout of 30000ms exceeded (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:27:20,374] INFO Submitting global closeSession request for session 0x30250d64f6e0001 (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:27:25,931] WARN Unable to read additional data from client sessionid 0x2025090352a0002, likely client has closed socket (org.apache.zookeeper.server.NIOServerCnxn)
[2020-10-29 00:27:30,062] WARN Unable to read additional data from client sessionid 0x2025090352a0000, likely client has closed socket (org.apache.zookeeper.server.NIOServerCnxn)
[2020-10-29 00:27:35,538] WARN Unable to read additional data from client sessionid 0x102508ff4380002, likely client has closed socket (org.apache.zookeeper.server.NIOServerCnxn)
[2020-10-29 00:27:40,026] WARN Unable to read additional data from client sessionid 0x2025090352a0002, likely client has closed socket (org.apache.zookeeper.server.NIOServerCnxn)
[2020-10-29 00:27:43,642] WARN Unable to read additional data from client sessionid 0x2025090352a0000, likely client has closed socket (org.apache.zookeeper.server.NIOServerCnxn)
[2020-10-29 00:27:51,346] WARN Unable to read additional data from client sessionid 0x102508ff4380002, likely client has closed socket (org.apache.zookeeper.server.NIOServerCnxn)
[2020-10-29 00:27:52,374] INFO Expiring session 0x10250d8e9830001, timeout of 30000ms exceeded (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:27:52,375] INFO Submitting global closeSession request for session 0x10250d8e9830001 (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:27:56,871] WARN Unable to read additional data from client sessionid 0x2025090352a0002, likely client has closed socket (org.apache.zookeeper.server.NIOServerCnxn)
[2020-10-29 00:27:59,895] WARN Unable to read additional data from client sessionid 0x2025090352a0000, likely client has closed socket (org.apache.zookeeper.server.NIOServerCnxn)
[2020-10-29 00:28:04,849] WARN Unable to read additional data from client sessionid 0x102508ff4380002, likely client has closed socket (org.apache.zookeeper.server.NIOServerCnxn)
[2020-10-29 00:28:11,575] WARN Unable to read additional data from client sessionid 0x2025090352a0002, likely client has closed socket (org.apache.zookeeper.server.NIOServerCnxn)
[2020-10-29 00:28:18,230] WARN Unable to read additional data from client sessionid 0x2025090352a0000, likely client has closed socket (org.apache.zookeeper.server.NIOServerCnxn)
[2020-10-29 00:28:18,414] WARN Unable to read additional data from client sessionid 0x102508ff4380002, likely client has closed socket (org.apache.zookeeper.server.NIOServerCnxn)
[2020-10-29 00:28:22,375] INFO Expiring session 0x30250d64f6e0002, timeout of 30000ms exceeded (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:28:22,375] INFO Submitting global closeSession request for session 0x30250d64f6e0002 (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:28:25,287] WARN Unable to read additional data from client sessionid 0x2025090352a0002, likely client has closed socket (org.apache.zookeeper.server.NIOServerCnxn)
[2020-10-29 00:28:32,271] WARN Unable to read additional data from client sessionid 0x102508ff4380002, likely client has closed socket (org.apache.zookeeper.server.NIOServerCnxn)
[2020-10-29 00:28:39,091] WARN Unable to read additional data from client sessionid 0x2025090352a0002, likely client has closed socket (org.apache.zookeeper.server.NIOServerCnxn)
[2020-10-29 00:28:40,266] WARN Unable to read additional data from client sessionid 0x2025090352a0000, likely client has closed socket (org.apache.zookeeper.server.NIOServerCnxn)
[2020-10-29 00:28:45,958] WARN Unable to read additional data from client sessionid 0x102508ff4380002, likely client has closed socket (org.apache.zookeeper.server.NIOServerCnxn)
[2020-10-29 00:28:54,764] WARN Unable to read additional data from client sessionid 0x2025090352a0002, likely client has closed socket (org.apache.zookeeper.server.NIOServerCnxn)
[2020-10-29 00:28:56,375] INFO Expiring session 0x10250d8e9830002, timeout of 30000ms exceeded (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:28:56,375] INFO Submitting global closeSession request for session 0x10250d8e9830002 (org.apache.zookeeper.server.ZooKeeperServer)
[2020-10-29 00:28:56,931] WARN Unable to read additional data from client sessionid 0x2025090352a0000, likely client has closed socket (org.apache.zookeeper.server.NIOServerCnxn)
[2020-10-29 00:28:59,554] WARN Unable to read additional data from client sessionid 0x102508ff4380002, likely client has closed socket (org.apache.zookeeper.server.NIOServerCnxn)