We are implementing a real-time aggregation process with Spark 2.2 structured streaming. The windows calculated can be based on event time that comes as an attribute on the object being processed in the stream and also can be calculated according to the processing time of the system like was possible in previous versions.
In addition we need to implement duplicate check to prevent the processing of duplicate events.
In the documentation it is stated that
Without watermark - Since there are no bounds on when a duplicate record may arrive, the query stores the data from all the past records as state. (https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#streaming-deduplication)
The problem is that looks like this state can grow forever and cause performance and memory size issues. There seems to be no way in the API to control it
The question is is there a way to implement duplicate check on events that do not have a notion of event time and still maintain control on the duplicate check sate like clearing it etc.
Thanks!