Spark: how is a window-based processing splitted to executors?

user_1943432 0
Viewed 377

I have a clear view of how Spark splits data to partitions within executors, then each partition is processed and then aggregated until a final "logical" dataframe.

However using windows, I feel like each window data should be in a single partition, so that each executor have all its data locally? Or is data still splitted, and then aggregated with a kind of magic?

An example of such Window is:

val window = Window
  .partitionBy("partition-col")
  .orderBy("order-col")
  .rowsBetween(Window.unboundedPreceding, Window.unboundedFollowing)
myDF.withColumn("sum", sum("aCol").over(window))

How does Spark handle this? How performant is it to use windows?

What if I process, let's say 50 columns from a Window? May it generate a lot of network exchange, or will each window be processed locally?

bigdata apache-spark-sql apache-spark
1 Answers

To calculate window functions Spark needs to arrange data so that values of columns/expressions mentioned in partitionBy are grouped in a single partition - as you expected.

For example try running a function with a window spanning entire dataframe. You will get following warning:

scala> df.withColumn("rn", row_number().over(Window.orderBy(lit(1)))).show
19/10/16 00:08:03 WARN WindowExec: No Partition Defined for Window operation! Moving all data to a single partition, this can cause serious performance degradation.
user_28900930
Related