How do you find out exactly what had caused the high GC time for the spark tasks in any given spark stage?

user_15026941 0
Viewed 1680

I do have a spark application where in one of the spark stage took most of the time 2.5hrs + . I did a dive deep and found for majority of the tasks the GC time was pretty high 60% of total task execution time.

The question that I have is :

  1. How do i co-relate this piece of spark task with my code ? enter image description here

  2. How do I identify what part of my spark code written using PySpark had caused the high GC time ? enter image description here

  3. In general what causes high GC time for any given spark task , I want to know ?

apache-spark
2 Answers

High GC means frequent GC or GC taking long time. A few suggestions with the limited info I could gather from the screenshots:

  1. One thing to check is are you caching big rdd/rdd's. Uncaching them as soon as they are no longer required will reduce the memory pressure. Is stage 68 part of first job, uncache unrequired data from previous jobs?

  2. How to figure out which operation is this: Use DAG visualization link on the top of stage,job pages to understand the flow. For SQL use SQL tab on the UI.

  3. Also there are 2000 tasks for ~ 40GB of shuffle data, each task handling 20 MB which is very small. better to have atleast ~128MB per task. tune this parallelism back to default 200 ?

  4. If you can't optimize your code then, use more memory by adding more nodes or nodes with larger memory.

user_28461680

From experience, high GC time is caused by tasks requiring more than the available memory. High GC time is often also accompanied by the tasks spilling to disk (entries in the Memory Spill and Disk Spill columns).

Also, from Learning Spark:

A high GC time signals too many objects on the heap (your executors may be memory-starved).

Damji, Jules S.,Wenig, Brooke,Das, Tathagata,Lee, Denny.

In my experience, a good mitigation is to increase the number of partitions read by the given stage to reduce the memory required by the individual tasks e.g. by decreasing spark.files.maxPartitionBytes when reading files, or increasing spark.sql.shuffle.partitions when joining dataframes.

user_18593700
Related