I have many spark dataframes on which I need to do the following:
1) load a single spark dataframe
2) select rows from it
3) merge it with all of the previous spark dataframes
Now, each of the above operations requires a different numberof partitions. The selecting rows requires many partitions, like 100 partitions. The merging requires very few partitions, like 10 partitions.
So, I really want it to work like this:
1) load a single spark dataframe
1.5) repartition into 100 partitions
2) select rows from it
2.5) repartition into 10 partitions
3) merge it with all of the previous spark dataframes
Now, how do I force this to repartition in between steps 1 and 2 and in between 2 and 3?
I know that when I call data = data.repartition(7) it is lazily evaluated, and so it only repartitions when it is actually saving.
So, I have been doing it like this:
1) load a single spark dataframe
1.5) repartition into 100 partitions
1.75) `df.count()` *just* to force materialization
2) select rows from it
2.5) repartition into 10 partitions
2.75) `df.count()` *just* to force materialization
3) merge it with all of the previous spark dataframes
Is there a better way to force it to repartition in between here? Is there a better way than running count() on the dataframe?