We are using Spark 2.4 to process around 445 GB of data. Our cluster had 150 workers, 7 CPU & 127 GB on each worker. Spark is deployed on standalone mode. Below is our config: one executor per worker with 7 CPU and 120 GB allocated. 2000 partitions in RDD.
I see some times jobs are failing due to executor loss. Below are the errors:
Driver logs:
ExecutorLostFailure (executor 82 exited caused by one of the running tasks) Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.\nDriver
Executor logs :
2020-07-03 01:53:10 INFO Worker:54 - Executor app-20200702155258-0011/13 finished with state EXITED message Command exited with code 137 exitStatus 137
[ERROR] 2020-06-29 17:34:42,074 SIGTERM handler org.apache.spark.executor.CoarseGrainedExecutorBackend - {} - RECEIVED SIGNAL TERM
- Started 161 remote fetches in 3 ms
[ERROR] 2020-06-29 17:34:42,079 SIGTERM handler org.apache.spark.executor.CoarseGrainedExecutorBackend - {} - RECEIVED SIGNAL TERM
687.0 in stage 87.0 (TID 45809). 66993 bytes result sent to driver
Also i noticed one worker box was down due to OOM error but i don't see any OOM errors in driver or workers logs.
Earlier i had tried running 2 executor per worker with 3 CPU, 60 GB per executor and 2000 partitions in RDD but then also same executor loss issue.
I think we have enough memory allocated to executors but then also executor are getting lost with code 137, which is think is due to memory issue but no OOM exception in driver or executor logs.
I tried updating "spark.memory.fraction" to "0.8", i saw better success rate for job but then also it fails some times with above errors.