I am working on Spark Structured Streaming. My input stream data looks like this:
id | col0 | col1 | ...
Sometimes rows with the same id arrives during the span of ~5 minutes. I want to keep only the first one and drop the duplicate ones that come late. I also want to write my output every one minute, as soon as a new row with a new id arrives. There are dropDuplicate and withWatermark functions in Spark Streaming, but I think if I use watermark, Spark waits until the watermark expires, so it is not suitable for this use case. Is there any good way to achieve this?