Spark - Executor heartbeat timed out after X ms

user_3333617 0
Viewed 22962

My program read data from files in a directory, the size of these files is 5 GB. I apply many function to these data. I am running spark as standalone (local), on a virtual machine having 32 GB RAM.

The used command :

bin/spark-submit --class ripeatlasanalysis.AnalyseTraceroute     --master local --driver-memory 30G  SparkExample-lowprints-0.0.5-SNAPSHOT-jar-with-dependencies.jar  1517961600  1518393600 3600

the 1517961600 1518393600 3600 are the parameters of jar file.

Sometimes the program run without errors, sometimes not, and the gotten error :

Exception in thread "main" org.apache.spark.SparkException: Job aborted due 
 to stage failure: Task 0 in stage 2.0 failed 1 times, most recent   failure: Lost task 
0.0 in stage 2.0 (TID 119, localhost, executor driver):  
ExecutorLostFailure (executor driver exited caused by one of the running   tasks) 
Reason: Executor heartbeat timed out after 128839 ms
 Driver stacktrace:
   at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGSchedule  r$$failJobAndIndependentStages(DAGScheduler.scala:1887)
   at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1875)

The question already asked here without responses.

apache-spark
1 Answers

I didn't find much information about your program but in general this can happened due to network issue, or stuck in computation but you can do two stepts. First, repartition the dataframe you work in more numbers for example df.repartition(1000) or you can repartition based on join columns in case of join. You can also increase the maxResultsSize,

Second: you can increase the executor and network timeout.

--conf spark.network.timeout 10000000 --conf spark.executor.heartbeatInterval=10000000   --conf spark.driver.maxResultSize=4g
user_25163560
Related