Ranking in Spark Structured Streaming

user_14551196 0
Viewed 381

I have a data stream that consists of data like this.

{Student, Class, CurrentScore}

I want to use a sliding window to calculate the statistic of these events:

spark.readStream(...).withColumn("processingTime",..).
window(col("processingTime"), "30 minutes", "5 minutes"), col("class"))

But now i want to take out the top 3 students by each class in this window, but the Structured Streaming apparently does not support rank()

How can I solve this problem ?

spark-structured-streaming apache-spark
1 Answers

Sorting operations (also rank()) are supported on streaming Datasets only after an aggregation and in Complete Output Mode; straight from the manuals.

Complete Output Mode is however not a realistic way of working. I would rather write to a location that is converted to a columnar format eventually, and then query and rank with an appropriate query tool.

user_69339930
Related