Apache Spark : Long Shuffle Read Blocked Time. Why?

user_10098090 0
Viewed 256

I have an 8-nodes Spark standalone cluster with a total of 880 GB RAM and 224 cores.

I just can't explain why the Shuffle Read Blocked Time is so long : about 20 minutes per task. Do you have an idea why ? What are the bottleneck in such case ?

To give more details, you can see bellow the details for stage :

details for stage apache Spark

The following tasks metrics from Spark Ui bellow :

Summary completed tasks metrics from Spark UI

The agregation per executor bellow :

Agregation per executor metrics from Spark UI

And the stage full DAG :

enter image description here

The executor tab :

enter image description here

The list of stages :

enter image description here

Thank you!

databricks apache-spark-sql apache-spark
0 Answers
Related