Kafka S3 Sink Connector - how to mark a partition as complete

user_2388940 0
Viewed 238

I am using Kafka sink connector to write data from Kafka to s3. The output data is partitioned into hourly buckets - year=yyyy/month=MM/day=dd/hour=hh. This data is used by a batch job downstream. So, before starting the downstream job, I need to be sure that no additional data will arrive in a given partition once the processing for that partition has started.

What is the best way to design this? How can I mark a partition as complete? i.e. no additional data will be written to it once marked as complete.

EDIT: I am using RecordField as timestamp.extractor. My kafka messages are guaranteed to be sorted within partitions by the partition field

system-design hadoop-partitioning batch-processing apache-kafka apache-spark
1 Answers

Depends on which Timestamp Extractor you are using in the Sink config.

You would have to guarantee the no records can have a timestamp earlier than the time you consume it.

AFAIK, the only way that's possible is using the WallClock Timestamp Extractor. Otherwise, you are consuming a Kafka Record timestamp, or some timestamp within each message. Both of which can be overwritten on the Producer end to some event in the past

user_23086830
Related