Confused about Kafka Consumer Properties

user_9322772 0
Viewed 139

I'm a bit confused by some of the consumer API configuration properties. It seems as though they either conflict, or cancel each other out. Can someone help me understand the difference between the following keys.

Definitions:

fetch.max.bytes: Maximum amount of data the server should return for a fetch request
max.partition.fetch.bytes: Max amount of data per-partition the server will return
max.poll.records: The maximum number of records returned in a single call to poll()

Example:
fetch.max.bytes: 30000 (30kb)
max.partition.fetch.bytes: 20000000 (20mb)
max.poll.records: 1000

To me it seems like the consumer definition above is saying it can accept up to 20mb of data/partition, but then only specifying max bytes of 30kb which doesn't make sense. Max poll records also seems to limit data intake since it's possible 1000 is too low or too high based on the size of each record.

kafka-consumer-api apache-kafka java
1 Answers

fetch.max.bytes and max.partition.fetch.bytes are fields of Fetch requests sent to Kafka brokers. They respectively determine the maximum size of the Fetch response the broker will send and the maximum size of data per partition the broker can return. It's the broker that uses these values to compute a Fetch response.

On the other hand, max.poll.records is a client-side only configuration. It determines how many records a call to poll() can return.

The consumer will fetch records in the background and buffer them so records are ready when poll() is called.

These settings allow for example to fetch records in batches, which is more efficient, but still pass them to the Consumer application in small chunks or even individually depending on the processing its doing.

user_17651890
Related