Is it possible to make a Kafka Consumer override/ignore its configurations when doing a records poll?

Liverbird97 0
Viewed 11

I have a Kafka consumer that should consume minimum 1MB worth of records at each poll. This data is then written to file and stored partioned by date - in example, records consumed during 2022.09.22 should be written to a file and stored to the date_id=20220922 folder. The file size should be a minimum of 1MB.

The configuration properties fetch.min.bytes and fetch.max.wait.ms are tuned to get the desired behavior. The problem, though, arrives when a new day occurs. On a day change, the consumer should consume the remaining records on topic (it is less than 1MB) without having to wait for the poll size threshold to be met or for the wait time to time out. The consumer should do a type of "force fetch" of the remaining records available on topic.

Is it possible to override the configuration of the consumer to achieve this behavior?

apache-kafka
1 Answers

The properties are what they are - you cannot change them at runtime without stopping the consumer and creating a new one with other config settings.

Worth mentioning that the HDFS/S3 sink connectors from Confluent already have a Date directory partitions. They also work for local storage, but distributed storage makes more sense when your kafka consumers are distributed

OneCricketeer0
Related