The meaning of the content of the __consumer_offsets topic

user_244413 0
Viewed 165

I am using kafka-dump-log tool to see the content of the __consumer_offsets topic. Can someone explain what does that content mean exactly ( offset, createtime, isvalid, keysize, valuesize, compresscodec, producerId, producerEpoch, isTransactional, headerKeys, key, offset, payload ) ?

[root@localhost kafka_2.11-2.0.0]# bin/kafka-dump-log.sh --files data/kafka/__consumer_offsets-32/00000000000000036360.log --value-decoder-class "kafka.serializer.StringDecoder" --offsets-decoder
    Dumping data/kafka/__consumer_offsets-32/00000000000000036360.log
    Starting offset: 36360
    offset: 36360 position: 0 CreateTime: 1623678492136 isvalid: true keysize: 19 valuesize: 28 magic: 2 compresscodec: NONE producerId: -1 producerEpoch: -1 sequence: 0 isTransactional: false headerKeys: [] key: offset::b0147:test:48 payload: 35
    offset: 36361 position: 0 CreateTime: 1623678492136 isvalid: true keysize: 19 valuesize: 28 magic: 2 compresscodec: NONE producerId: -1 producerEpoch: -1 sequence: 1 isTransactional: false headerKeys: [] key: offset::b0147:test:64 payload: 34
    offset: 36362 position: 0 CreateTime: 1623678492136 isvalid: true keysize: 19 valuesize: 28 magic: 2 compresscodec: NONE producerId: -1 producerEpoch: -1 sequence: 2 isTransactional: false headerKeys: [] key: offset::b0147:test:14 payload: 34
    ...
    offset: 36460 position: 5497 CreateTime: 1623678503139 isvalid: true keysize: 9 valuesize: 24 magic: 2 compresscodec: NONE producerId: -1 producerEpoch: -1 sequence: -1 isTransactional: false headerKeys: [] key: {"metadata":"b0147"} payload: {"protocolType":"consumer","protocol":null,"generationId":2,"assignment":"{}"}
    offset: 36461 position: 5598 CreateTime: 1623678554656 isvalid: true keysize: 19 valuesize: -1 magic: 2 compresscodec: NONE producerId: -1 producerEpoch: -1 sequence: -1 isTransactional: false headerKeys: []
    ...
apache-kafka
1 Answers

The fields you see are not specific to __consumer_offsets, they are just the metadata about each message/batch in the log and you will get them for all topics, see https://kafka.apache.org/documentation/#recordbatch

If you want to explore the content of __consumer_offsets you need to know that it contains 2 types of records:

  • OffsetMetadata: this is information about the offsets committed by groups
  • GroupMetadata: this is information about members of consumer groups

To show OffsetMetadata, you can use:

./bin/kafka-console-consumer.sh \
  --formatter "kafka.coordinator.group.GroupMetadataManager\$OffsetsMessageFormatter" \
  --bootstrap-server localhost:9092 --topic __consumer_offsets

To show GroupMetadata, you can use:

./bin/kafka-console-consumer.sh \
  --formatter "kafka.coordinator.group.GroupMetadataManager\$GroupMetadataMessageFormatter" \
  --bootstrap-server localhost:9092 --topic __consumer_offsets
user_17651890
Related