I am getting aggregated data in event-time windows, but the data I need is the one contained in the newest 3 windows. What would be the right way to erase the other processed older windows I don need?
val query = lines
.withWatermark("timestamp", "20 seconds")
.groupBy(window($"timestamp", "10 seconds"),$"value").count().sort(-$"window")
.writeStream
.outputMode("complete")
.format("console")
.start()
I would like to see only the last 3 newest windows. Would filtering out the windows older than 30 seconds good enough?
Thanks