I am pretty new to Apache Spark Structured Streaming and I am doing a basic POC. My requirement is to build a rule engine which will read data from HDFS which is receiving a continous stream of data from multiple sources. I need to run different rules on the basis of configurable time window on this HDFS data. E.g. there could be one rule which needs to run on last 5 minutes of data and second rule will run on last 7 minutes of data.
If I create two streams, one with a frequency of 5 minutes and other with a frequency of 7 minutes, since spark is an in-memory engine, is it possible that the stream with 7 minutes frequency read the data first from other stream (5 minutes frequency) and only read the missing window data from HDFS? Or it will re-read all the data from HDFS?
If both the streams will read complete data from HDFS independently, then latency of processing increases and data duplication is also there in reads. Any other mechanism or framework available that can avoid this duplicate reads?