As you can see, I have multiple window functions(say count/sum etc.) being executed in stages(2,3,4,5,....). I may have around 15-20 such stages containing window functions. Given a huge dataframe df (in GBs), each window function add a new column to it. In the end I want the original df plus each of these new columns in a new combined dataframe. Right now, these functions are adding the new column sequentially and hence it take a long time for the whole job to finish.
To make this faster, obvious parallel execution comes to my mind:
- From each window function, I can write the output dataframe(with 1 new column) to AWS S3(or disk) and then join each of those dataframe on a unique key in the end:
listOfComputationsToPerform.par.foreach{
Apply window function & compute new_dataframe
write new_dataframe to S3/disk
}
Read each dataframe again from S3 and perform joins on a unique key to compute 1 final dataframe
- We can save time if we don't do the write & read from S3/disk above but I am not sure if all these dataframes running on different executors can be joined in the end without running into any memory issues:
df_list=List[]
listOfComputationsToPerform.par.foreach{
compute new_dataframe => // code to be run in parallel for each i}
df_list = df_list :+ new_dataframe
df_list.reduce(_.join(_, Seq("id")))
I am wondering which one out of 1 and 2 is a better way to go about this. Any other suggestions are welcome as well.
