Distinct operation in spark structured streaming with a window operation

Viewed 31

I want to implement a distinct operation in Spark structured code. I have already watermarked it and windowed it, but still, Spark is not able to execute it. FYI - Distinct comes under the list of unsupported operations in Spark streaming, but I feel that distinct in a window is a very basic operation to not be supported by Spark. Reference

Snippet:

    val fieldLocationJobDF = eligibleJobDF
      .withWatermark("timestamp", "24 hours")
      .groupBy(window(col("timestamp"), "15 minutes"), col(JOB_ID), col(JOB_LOCATION_TYPE))
      .agg(col(JOB_ID), col(JOB_LOCATION_TYPE))
      .distinct()

Am I doing anything wrong here?

_

0 Answers
Related