I am using kafka 1.1.0, single consumer and I get this error though my poll is very quick, one poll on every 100 milli seconds elapsed. I use manual commit and duration between two commit request may change (can be even hours). But not poll.
I still get this error "The coordinator is not aware of this member." with message
org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records.
What could be the reason for this error. I have searched all over the place but I couldn't get any information in my context of problem.
Both "max.poll.interval.ms" and session timeout are default.
From logs I can see last successful commit was at "11:20:11.677" and failed commit was at "11:20:35.742" just 24 seconds apart.
Its application process with its own group id. I can confirm there was no other process running with same group id.
This is from debug log:
2018-11-17 17:03:39.227 - DEBUG o.a.k.c.c.i.AbstractCoordinator Attempt to heartbeat failed for since member id consumer-1-e237c8ed-fed7-4687-be4d-99e333e48af3 is not valid.
2018-11-17 17:03:39.227 - DEBUG o.a.k.c.c.i.AbstractCoordinator Disabling heartbeat thread 2018-11-17 17:08:09.241 - DEBUG o.apache.kafka.clients.NetworkClient Sending metadata request (type=MetadataRequest, topics=) to node 10.0.10.220:9092 (id: 1 rack: null)
2018-11-17 17:08:09.241 - DEBUG org.apache.kafka.clients.Metadata - Updated cluster metadata version 505 to Cluster(id = 5zFDtyruTP-cIctUkxG0Pg, nodes = [10.0.10.222:9092 (id: 3 rack: null), 10.0.10.220:9092 (id: 1 rack: null), 10.0.10.221:9092 (id: 2 rack: null)], partitions = [])
2018-11-17 17:12:09.472 - DEBUG o.apache.kafka.clients.NetworkClient Node 3 disconnected.
2018-11-17 17:12:09.472 - DEBUG o.apache.kafka.clients.NetworkClient Sending metadata request (type=MetadataRequest, topics=) to node 10.0.10.220:9092 (id: 1 rack: null)
2018-11-17 17:12:09.472 - DEBUG org.apache.kafka.clients.Metadata - Updated cluster metadata version 506 to Cluster(id = 5zFDtyruTP-cIctUkxG0Pg, nodes = [10.0.10.222:9092 (id: 3 rack: null), 10.0.10.221:9092 (id: 2 rack: null), 10.0.10.220:9092 (id: 1 rack: null)], partitions = [])
2018-11-17 17:17:09.502 - DEBUG o.apache.kafka.clients.NetworkClient Initialize connection to node 10.0.10.222:9092 (id: 3 rack: null) for sending metadata request
2018-11-17 17:17:09.503 - DEBUG o.apache.kafka.clients.NetworkClient Initiating connection to node 10.0.10.221:9092 (id: 2 rack: null)
2018-11-17 17:17:09.503 - DEBUG o.apache.kafka.clients.NetworkClient Sending metadata request (type=MetadataRequest, topics=) to node 10.0.10.220:9092 (id: 1 rack: null)
2018-11-17 17:17:09.503 - DEBUG o.a.kafka.common.network.Selector Created socket with SO_RCVBUF = 32768, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node 2
2018-11-17 17:17:09.503 - DEBUG o.apache.kafka.clients.NetworkClient Completed connection to node 2. Fetching API versions.
2018-11-17 17:17:09.503 - DEBUG o.apache.kafka.clients.NetworkClient Initiating API versions fetch from node 2.
2018-11-17 17:17:09.503 - DEBUG org.apache.kafka.clients.Metadata - Updated cluster metadata version 507 to Cluster(id = 5zFDtyruTP-cIctUkxG0Pg, nodes = [10.0.10.220:9092 (id: 1 rack: null), 10.0.10.221:9092 (id: 2 rack: null), 10.0.10.222:9092 (id: 3 rack: null)], partitions = [])
2018-11-17 17:17:09.503 - DEBUG o.apache.kafka.clients.NetworkClient Recorded API versions for node 3: (Produce(0): 0 to 5 [usable: 5], Fetch(1): 0 to 7 [usable: 7], ListOffsets(2): 0 to 2 [usable: 2], Metadata(3): 0 to 5 [usable: 5], LeaderAndIsr(4): 0 to 1 [usable: 1], StopReplica(5): 0 [usable: 0], UpdateMetadata(6): 0 to 4 [usable: 4], ControlledShutdown(7): 0 to 1 [usable: 1], OffsetCommit(8): 0 to 3 [usable: 3], OffsetFetch(9): 0 to 3 [usable: 3], FindCoordinator(10): 0 to 1 [usable: 1], JoinGroup(11): 0 to 2 [usable: 2], Heartbeat(12): 0 to 1 [usable: 1], LeaveGroup(13): 0 to 1 [usable: 1], SyncGroup(14): 0 to 1 [usable: 1], DescribeGroups(15): 0 to 1 [usable: 1], ListGroups(16): 0 to 1 [usable: 1], SaslHandshake(17): 0 to 1 [usable: 1], ApiVersions(18): 0 to 1 [usable: 1], CreateTopics(19): 0 to 2 [usable: 2], DeleteTopics(20): 0 to 1 [usable: 1], DeleteRecords(21): 0 [usable: 0], InitProducerId(22): 0 [usable: 0], OffsetForLeaderEpoch(23): 0 [usable: 0], AddPartitionsToTxn(24): 0 [usable: 0], AddOffsetsToTxn(25): 0 [usable: 0], EndTxn(26): 0 [usable: 0], WriteTxnMarkers(27): 0 [usable: 0], TxnOffsetCommit(28): 0 [usable: 0], DescribeAcls(29): 0 [usable: 0], CreateAcls(30): 0 [usable: 0], DeleteAcls(31): 0 [usable: 0], DescribeConfigs(32): 0 to 1 [usable: 1], AlterConfigs(33): 0 [usable: 0], AlterReplicaLogDirs(34): 0 [usable: 0], DescribeLogDirs(35): 0 [usable: 0], SaslAuthenticate(36): 0 [usable: 0], CreatePartitions(37): 0 [usable: 0], CreateDelegationToken(38): 0 [usable: 0], RenewDelegationToken(39): 0 [usable: 0], ExpireDelegationToken(40): 0 [usable: 0], DescribeDelegationToken(41): 0 [usable: 0], DeleteGroups(42): 0 [usable: 0])
2018-11-17 17:17:09.503 - DEBUG o.apache.kafka.clients.NetworkClient Completed connection to node 2. Fetching API versions.
2018-11-17 17:17:09.503 - DEBUG o.apache.kafka.clients.NetworkClient Initiating API versions fetch from node 2.
2018-11-17 17:17:09.503 - DEBUG org.apache.kafka.clients.Metadata - Updated cluster metadata version 507 to Cluster(id = 5zFDtyruTP-cIctUkxG0Pg, nodes = [10.0.10.220:9092 (id: 1 rack: null), 10.0.10.221:9092 (id: 2 rack: null), 10.0.10.222:9092 (id: 3 rack: null)], partitions = [])
2018-11-17 17:17:09.503 - DEBUG o.apache.kafka.clients.NetworkClient Recorded API versions for node 3: (Produce(0): 0 to 5 [usable: 5], Fetch(1): 0 to 7 [usable: 7], ListOffsets(2): 0 to 2 [usable: 2], Metadata(3): 0 to 5 [usable: 5], LeaderAndIsr(4): 0 to 1 [usable: 1], StopReplica(5): 0 [usable: 0], UpdateMetadata(6): 0 to 4 [usable: 4], ControlledShutdown(7): 0 to 1 [usable: 1], OffsetCommit(8): 0 to 3 [usable: 3], OffsetFetch(9): 0 to 3 [usable: 3], FindCoordinator(10): 0 to 1 [usable: 1], JoinGroup(11): 0 to 2 [usable: 2], Heartbeat(12): 0 to 1 [usable: 1], LeaveGroup(13): 0 to 1 [usable: 1], SyncGroup(14): 0 to 1 [usable: 1], DescribeGroups(15): 0 to 1 [usable: 1], ListGroups(16): 0 to 1 [usable: 1], SaslHandshake(17): 0 to 1 [usable: 1], ApiVersions(18): 0 to 1 [usable: 1], CreateTopics(19): 0 to 2 [usable: 2], DeleteTopics(20): 0 to 1 [usable: 1], DeleteRecords(21): 0 [usable: 0], InitProducerId(22): 0 [usable: 0], OffsetForLeaderEpoch(23): 0 [usable: 0], AddPartitionsToTxn(24): 0 [usable: 0], AddOffsetsToTxn(25): 0 [usable: 0], EndTxn(26): 0 [usable: 0], WriteTxnMarkers(27): 0 [usable: 0], TxnOffsetCommit(28): 0 [usable: 0], DescribeAcls(29): 0 [usable: 0], CreateAcls(30): 0 [usable: 0], DeleteAcls(31): 0 [usable: 0], DescribeConfigs(32): 0 to 1 [usable: 1], AlterConfigs(33): 0 [usable: 0], AlterReplicaLogDirs(34): 0 [usable: 0], DescribeLogDirs(35): 0 [usable: 0], SaslAuthenticate(36): 0 [usable: 0], CreatePartitions(37): 0 [usable: 0], CreateDelegationToken(38): 0 [usable: 0], RenewDelegationToken(39): 0 [usable: 0], ExpireDelegationToken(40): 0 [usable: 0], DescribeDelegationToken(41): 0 [usable: 0], DeleteGroups(42): 0 [usable: 0]) 2018-11-17 17:17:09.503 - DEBUG o.apache.kafka.clients.NetworkClient Recorded API versions for node 2: (Produce(0): 0 to 5 [usable: 5], Fetch(1): 0 to 7 [usable: 7], ListOffsets(2): 0 to 2 [usable: 2], Metadata(3): 0 to 5 [usable: 5], LeaderAndIsr(4): 0 to 1 [usable: 1], StopReplica(5): 0 [usable: 0], UpdateMetadata(6): 0 to 4 [usable: 4], ControlledShutdown(7): 0 to 1 [usable: 1], OffsetCommit(8): 0 to 3 [usable: 3], OffsetFetch(9): 0 to 3 [usable: 3], FindCoordinator(10): 0 to 1 [usable: 1], JoinGroup(11): 0 to 2 [usable: 2], Heartbeat(12): 0 to 1 [usable: 1], LeaveGroup(13): 0 to 1 [usable: 1], SyncGroup(14): 0 to 1 [usable: 1], DescribeGroups(15): 0 to 1 [usable: 1], ListGroups(16): 0 to 1 [usable: 1], SaslHandshake(17): 0 to 1 [usable: 1], ApiVersions(18): 0 to 1 [usable: 1], CreateTopics(19): 0 to 2 [usable: 2], DeleteTopics(20): 0 to 1 [usable: 1], DeleteRecords(21): 0 [usable: 0], InitProducerId(22): 0 [usable: 0], OffsetForLeaderEpoch(23): 0 [usable: 0], AddPartitionsToTxn(24): 0 [usable: 0], AddOffsetsToTxn(25): 0 [usable: 0], EndTxn(26): 0 [usable: 0], WriteTxnMarkers(27): 0 [usable: 0], TxnOffsetCommit(28): 0 [usable: 0], DescribeAcls(29): 0 [usable: 0], CreateAcls(30): 0 [usable: 0], DeleteAcls(31): 0 [usable: 0], DescribeConfigs(32): 0 to 1 [usable: 1], AlterConfigs(33): 0 [usable: 0], AlterReplicaLogDirs(34): 0 [usable: 0], DescribeLogDirs(35): 0 [usable: 0], SaslAuthenticate(36): 0 [usable: 0], CreatePartitions(37): 0 [usable: 0], CreateDelegationToken(38): 0 [usable: 0], RenewDelegationToken(39): 0 [usable: 0], ExpireDelegationToken(40): 0 [usable: 0], DescribeDelegationToken(41): 0 [usable: 0], DeleteGroups(42): 0 [usable: 0])
I see member id not unique message in debug and kafka consumer client keeps retrying for connection.