I am streaming data like this: time, id, value
I want to keep only one record for each id, with the newest value. What is the best way to deal with this problem?
Prefer to use Pyspark
Keep only the newest data with Spark structured streaming
Viewed 103
1 Answers
from pyspark.sql import Window
from pyspark.sql.functions import rank, col, monotonically_increasing_id
window = Window.partitionBy("id").orderBy("time",'tiebreak')
df_s
.withColumn('tiebreak', monotonically_increasing_id())
.withColumn('rank', rank().over(window))
.filter(col('rank') == 1).drop('rank','tiebreak')
.show()
Rank and tiebreaks are added to remove duplicates or ties across and within window partitions.
Related