In Apache Kafka, who keeps track of the last message read by the consumer? And also who keeps track of which consumer group ID reads from which partition? Is all this information is in the zookeeper?
1 Answers
Every consumer group maintains its offset per topic partition. Since v0.9 the information of committed offsets for every consumer group is stored in this internal topic (prior to v0.9 this information was stored on Zookeeper). When the offset manager receives an OffsetCommitRequest, it appends the request to a special compacted Kafka topic named __consumer_offsets. Finally, the offset manager will send a successful offset commit response to the consumer, only when all the replicas of the offsets topic receive the offsets.
Regarding your question about partition assignment, Kafka uses partition.assignment.strategy to determine how partitions are assigned to consumers. This propertydefaults to RangeAssignor:
The range assignor works on a per-topic basis. For each topic, we lay out the available partitions in numeric order and the consumers in lexicographic order. We then divide the number of partitions by the total number of consumers to determine the number of partitions to assign to each consumer. If it does not evenly divide, then the first few consumers will have one extra partition. For example, suppose there are two consumers C0 and C1, two topics t0 and t1, and each topic has 3 partitions, resulting in partitions t0p0, t0p1, t0p2, t1p0, t1p1, and t1p2. The assignment will be: C0: [t0p0, t0p1, t1p0, t1p1] C1: [t0p2, t1p2]
The other two options are RoundRobinAssignor and StickyAssignor
Related