Spark Stage performance, found GC Time very high just for few tasks

user_1810431 0
Viewed 4303

I'm trying to tune a Spark application, in order to reduce overall time execution, but I'm having a strange behaviour during a Stage execution.

Basically just 14/120 tasks needs around 20 min to finish, the others instead take 4 or 5 min to be completed.

Looking a the Spark UI, the partitioning seems good, the only difference I see is the GC Time that is very high for the 14 tasks.

I attach an image of the situation.

Do you have any idea for find the performance solution? enter image description here enter image description here

rdd apache-spark garbage-collection performance
1 Answers

I had a similar problem and could resolve it by using Parallel GC instead of G1GC. You may add the following options to the executors additional Java options in the submit request

-XX:+UseParallelGC -XX:+UseParallelOldGC
user_11514720
Related