What happen if I use reduceByKey or groupByKey one large datasets having only two keys

user_6579116 0
Viewed 321

I'm using spark to process my data. I have paired RDD that is distributed on multiple executors. The data size is 10tb and the no of partitions are 4000. The total no of executors are 100 and the memory on every executor is 20GB.

So the size of every partition will be around 128mb.

i.e

pairedRDD = sc.textFile("myhost:9000/data/largedataset.txt", 4000);
resultRdd = pairedRDD.reduceByKey(lambda (x, y) : (x + y));

The data in the file is like,

1 Apple
1 Apple
2 Mango
1 Apple
3 Banana

We have only 3 keys. So all the related keys will move to same partition. As we have one partition size 512mb and we have 4000 partition, it will cause any OOM error or it will execute without any error.

If it will execute without any error then how Spark is handling the size of data internally?

rdd pyspark apache-spark
1 Answers

The behavior of Spark will heavily depend on what you choose: reduceByKey or groupByKey.

  • GroupByKey will bring all data in one partition (using shuffling) and in your case it will cause OOM (Out Of Memory).
  • ReduceByKey will first do logic on each executor and then do bring the resulted values on one executor.
user_12091590
Related