Spark GC time is very high causing task execution slow

Question

I am running spark application on 5 node cluster. Each node have configuration as 16 core and 64 GB ram.

There are 4 stages in my application. 3 stages executed fast for 4th stage ie mapToPair is it taking too much time(almost 4 hours).

Below is the job statistics for job

I have checked executors logs as well, didn't see any errors/exception. Could you please suggest me if any configuration changes needs to be done to improve the performance?

user_3344461 · Answer

You can try to repartition the dataframe inside the application and increase cores, with partitons = cores and check again.

user_2990232 · Answer

https://spark.apache.org/docs/latest/tuning.html

first try
.config("spark.executor.memory","8g")

http://www.oracle.com/technetwork/articles/java/g1gc-1984535.html

Spark GC time is very high causing task execution slow

2 Answers