Where does zookeeper store kafka cluster and related information?

user_4820101 0
Viewed 910

By saying cluster info, I am referring to information like

  1. subscribed consumers/consumer groups
  2. read and committed offsets
  3. leaders and followers of a partition
  4. topics on the server etc.

Does zookeeper keep this info in its own db (though I never heard of zookeeper having any db of its own till date) or it stores this info in the Kafka cluster on some topics etc?

EDIT: and a follow up Q: How does Zookeeper retrive the consumer offsets from __consumer_offsets topic?

kafka-topic kafka-consumer-api apache-kafka apache-zookeeper
1 Answers

The ZooKeeper Data Model ZooKeeper has a hierarchal name space, much like a distributed file system. The only difference is that each node in the namespace can have data associated with it as well as children. It is like having a file system that allows a file to also be a directory. Paths to nodes are always expressed as canonical, absolute, slash-separated paths; there are no relative reference.

So there is no database.

Offsets are kept in kafka, not in zookeeper, in this topic: __consumer_offsets

In a zookeeper node:

Gives the list of active brokers

ls /brokers/ids

Gives the list of topics

ls /brokers/topics

Gives more detailed information of the broker id '0'

get /brokers/ids/0

here you can read more about the kafka datastructures in zookeeper: Kafka datastructures zookeeper

There is a KIP for removing zookeeper alltogether: KIP Remove zookeeper

user_113190160
Related