Spark: executor heartbeat timed out

Question

I am working in a databricks cluster that have 240GB of memory and 64 cores. This the settings I defined.

import pyspark
from pyspark.sql import SparkSession
from pyspark.sql.types import *
import pyspark.sql.functions as fs
from pyspark.sql import SQLContext
from pyspark import SparkContext
from pyspark.sql.functions import count
from pyspark.sql.functions import col, countDistinct
from pyspark import SparkContext
from geospark.utils import GeoSparkKryoRegistrator, KryoSerializer
from geospark.register import upload_jars
from geospark.register import GeoSparkRegistrator
spark.conf.set("spark.sql.shuffle.partitions", 1000)
#Recommended settings for using GeoSpark
spark.conf.set("spark.driver.memory", "20g")
spark.conf.set("spark.network.timeout", "1000s")
spark.conf.set("spark.driver.maxResultSize", "10g")
spark.conf.set("spark.serializer", KryoSerializer.getName)
spark.conf.set("spark.kryo.registrator", GeoSparkKryoRegistrator.getName)
upload_jars()
SparkContext.setSystemProperty("geospark.global.charset","utf8")
spark.conf.set

I am working with large datasets and this is the error I get after hours of running.

org.apache.spark.SparkException: Job aborted due to stage failure: Task 3 in stage 10.0 failed 4 times, most recent failure: Lost task 3.3 in stage 10.0 (TID 6054, 10.17.21.12, executor 7): 

ExecutorLostFailure (executor 7 exited caused by one of the running tasks) Reason: Executor heartbeat timed out after 170684 ms

user_17672070 · Answer

Let the heartbeat Interval be default(10s) and increase the network time out interval(default -120 s) to 300s (300000ms) and see. Use set and get .

    spark.conf.set("spark.sql.", )
    spark.conf.set("spark.network.timeout", 300000 )

or run this script in the notebook .

    %scala
    dbutils.fs.put("dbfs:/databricks/init/set_spark_params.sh","""
    |#!/bin/bash
    |
    |cat << 'EOF' > /databricks/driver/conf/00-custom-spark-driver-defaults.conf 
    |[driver] {
    |  "spark.network.timeout" = "300000"
    |}
    |EOF
    """.stripMargin, true)

user_17663942 · Answer

The error tells you that the worker has timed out because it took too long. There is probably some bottleneck happening in the background. Check the spark UI for executor 7, task 3 and stage 10. You also want to check the query that you have been running.

You also want to check these setting for better configuration:

spark.conf.set("spark.databricks.io.cache.enabled", True) # delta caching
spark.conf.set("spark.sql.adaptive.skewJoin.enabled", True) # adaptive query execution for skewed data
spark.conf.set("spark.sql.autoBroadcastJoinThreshold", -1) # setting treshhold on broadcasting 
spark.conf.set("spark.databricks.optimizer.rangeJoin.binSize", 20) #range optimizer

Feel free to give us more info on the Spark UI, we can better help you find the problem that way. Also, what kind of query were you doing?

user_9599091 · Answer

Can you please try the following options ,

repartition the dataframe you work in more numbers for example df.repartition(1000)
--conf spark.network.timeout 10000000
--conf spark.executor.heartbeatInterval=10000000

Spark: executor heartbeat timed out

3 Answers