My partitioned data looks like the following
+---+-------+
|tag|count |
+---+-------+
|29 |240 |
|22 |9941280|
|27 |360 |
|26 |900 |
|23 |1941286|
|25 |32939 |
|24 |300148 |
+---+-------+
where I have partitioned the row according to their tag column. Now I am using mapPartition to process the partitioned data parallelly.
My setup contains 3 worker nodes and 1 master. I expected the partitions will be distributed among the workers so that each have approximately the same amount of data.
But when I check the logs I saw the following:
worker: 1
Partition Quartets Count: 1941286
Partition Quartets Count: 9941280
worker: 2
Partition Quartets Count: 360
Partition Quartets Count: 300148
worker: 3
Partition Quartets Count: 240
Partition Quartets Count: 900
Partition Quartets Count: 32939
Clearly, Apache Spark is distributing the partitions based on partition count, not partition size. Obviously, a better mapping would be something like: | 9941280 | 1941286 | 300148+32939+240+360+900 |
Is there any way of enforcing this?