Flink version: v1.15.2
I had a problem with Apache Flink: Flink failed to submit offset when the Kafka Source table of the Flink task used the same group ID as other Kafka consumers. The problem scenario is described as follows:
I have a Java application that is a Kafka consumer, using the consumer group 'TopicA' to consume data from the topic 'topic_a'
There is a Flink task, and the Kafka consumer group used by its Kafka Source table is also 'TopicA', but consumes the data of the topic 'topic_b'
At this point, the following error appears in the log information of the Flink task: Asynchronous auto-commit of offsets {topic_b-0=OffsetAndMetadata{offset=xxx, leaderEpoch=0, metadata=''}} failed: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing max.poll.interval.ms or by reducing the maximum size of batches returned in poll() with max.poll.records.