background:
I'm running a spark job on a huge cluster with heavy workloads that constantly has ill-state nodes, which receives task, respond driver's heartbeat, not actually working, and takes forever to run, and may fail finally so driver need to re-submit the task somewhere else.
what I did to deal with the ill-state nodes:
I'm setting spark.blacklist.enabled to True to make sure the re-submitted task goes somewhere else (and in blink of an eye job finished). However, as I found out in log, the blacklist is only working for one stage:
Blacklisting executor 28 for stage 0
so next stage there will certainly try the ill node again, and there's high chance that the ill node may not come back to normal. I just met such situation that a node keeps failing task for 48 hours 180 times and just kill itself finally.
18/11/11 19:47:26 WARN cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: Container marked as failed: container_1534870268016_1640615_01_000051 on host: ill-datanode. Exit status: -100. Diagnostics: Container released on a *lost* node
executor like this drags spark application's performance heavily.
so I comes up with plan B: I kill it myself
I found there are 2 function to manage the executor called SparkSession.sparkContext.killExecutor(executorId: String) and requestExecutors(numAdditionalExecutors: Int). But to remove executor using such function I must know which executor failed during last job.
How to do that?