Perform a batch validation in Kafka and sent to corresponding topic

user_1049104 0
Viewed 95

I have the following batch format stored in a Kafka topic:

Data generated -->  B2E, T3, T2, T1, B2S | B1E, T3, T2, T1, B1S  --> Data Consumed

Here BS indicate a batch start, BE indicates batch end and t1,t2,t3 is three different data.

Here is the scanerio:

  1. Before pushing batch end record B1E to the topic, we got to know that the B1 batch is invalid after doing some validation.

  2. In that case all data of B1 batch ranging BS to BE should go to a particular topic.

So in above example b1 batch should go to topic T1 and b2 batch should go to T2.

How can I do that using Kafka.?

confluent-platform apache-kafka
1 Answers

simple flow digram

According to the above picture, Read and validate each message by message and update some in the memory window (memory map with batch data) until the relevant batch end BE is received. After batch end BE received read relevant batch data from the window and publish all the batch records to the topic selected from processed (validated) results.

To windowing, you can use a memory map like <Key = BatchID> <Values = Array of Batch Data Object> or Kafka streaming state store.

If you need something like a KStream solution, It will be a stream flow like below

Kstream solution

user_108667980
Related