How to merge two kafka topics data when second topic data arrives late

user_3978986 0
Viewed 715

Have two kafka topics of two different tables from mysql database .

Table1 - Transaction Data

Table2 - Transaction Details Data

Now i need to merge data from these two kafka topics (aka mysql tables) and push to Mongo Db as one document.

Though i can use kafka streams to do the same, but need suggestions how to handle following cases

Case 1 - when Table1 data arrives but not Table2 data

Case 2 - when Table2 data arrives but not Table1 data

apache-kafka-streams apache-kafka
1 Answers

Store your data temporarily in a Windowed key-value store.

When data arrives from stream1: see if matching data from stream2 is available. If so, combine the data and store in MongoDB. If not, store the data for stream 1 in the windowed store.

When data arrives from stream2: see if matching data from stream1 is available. If so, combine the data and store it in MongoDB. If not, store the data from stream2 in the windowed store.

The default implementation of the windowed store in KafkaStreams is a RocksDB instance per partition. To make this work, you have to make sure both streams have the same partitions.

This is exactly what kafka streams does behind a KStream.join(Kstream, ...):

KStream<String, String> joined = left.join(right,
    (leftValue, rightValue) -> combine(leftValue, rightValue),
    JoinWindows.of(...),
    Joined.with(...)
);

The size of the join window is normally limited to avoid keeping data infinitely long. The limit should be the maximum difference between the different arrival times of data from the different streams.

user_10328900
Related