controlling the order in which the partitions are processed in spark

user_1700223 0
Viewed 47

So, for an optimal usage of my cluster CPU, I would like to have my partitions processed roughly in the order given by MyPartitioner.getPartition(key). I know they are processed in parallel and their processing order cannot be fully controlled but at least can I control the order in which the driver submits tasks to the executors ?

Something like :

 task 0 - process partition 0
 task 1 - process partition 1
 ...
 task n - process partition n
apache-spark-2.0 apache-spark
1 Answers

Long story short, One cannot control the processing order of partitions in spark. The only thing that one can do is to make sure the partitions are more or less equal

user_17002230
Related