Gracefully shutting down Kafka Consumer with static membership

user_4014778 0
Viewed 86

Given a Kafka Consumer with a static membership, what would be the best practice to gracefully shut it down?

From Kafka - the definitive guide: By calling .close()

Always close() the consumer before exiting. This will close the network connections and sockets. It will also trigger a rebalance immediately rather than wait for the group coordinator to discover that the consumer stopped sending heartbeats and is likely dead, which will take longer and therefore result in a longer period of time in which consumers can’t consume messages from a subset of the partitions.

So if a static membership is used, close() is not what we really want. Maybe a tweaked version if it to close the network sockets but not causing a rebalance?

Another thing to note is that the unsubscribe is not causing a rebalance.

I am really wondering how others imagine the shutdown mechanism in this special case.

confluent-platform kafka-consumer-api apache-kafka java
1 Answers

Not sure what are the special cases you are after, that you can include it in your note. Also, how long you want to shutdown your consumer in the runtime.

If you are using a static membership, the consumer group will not trigger rebalance unless

  • A new member joins
  • A leader re-joins (possibly due to topic assignment change)
  • An existing member offline time is over session timeout
  • Broker receives a leave group request containing a list of group.instance.ids

When you set a consumer as static member, it will not send leave group request when they go offline (in your case shutdown for a special case), which means we shall only rely on session.timeout to trigger group rebalance. Therefore, it is also advised to make the session timeout large enough so that broker side will not trigger rebalance too frequently due to member come and go.

I believe, close() will become obsolete here.

Also, it is important how we are handling the runtime exception scenarios (socket timeout, elapse of max.poll.interval, broker leader unavailability etc) irrespective of static or dynamic memberships, and when a consumer reaching the state of final block.

finally
{
  consumer.close();
}
user_105840610
Related