I have a hefty spark app that keeps retrying, and the only helpful logs I can find through the UI is this from stdout:
2021-08-16 17:30:52 ERROR TransportRequestHandler:293 - Error sending result RpcResponse{requestId=6321165190495215882, body=NioManagedBuffer{buf=java.nio.HeapByteBuffer[pos=0 lim=81 cap=156]}} to /192.168.56; closing connection
java.nio.channels.ClosedChannelException
at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source)
2021-08-16 17:30:52 ERROR TransportRequestHandler:293 - Error sending result RpcResponse{requestId=7370805066606093965, body=NioManagedBuffer{buf=java.nio.HeapByteBuffer[pos=0 lim=81 cap=156]}} to /192.168.56; closing connection
java.nio.channels.ClosedChannelException
at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source)
2021-08-16 17:30:52 ERROR TransportRequestHandler:293 - Error sending result RpcResponse{requestId=8523609779541081889, body=NioManagedBuffer{buf=java.nio.HeapByteBuffer[pos=0 lim=81 cap=156]}} to /192.168.56; closing connection
java.nio.channels.ClosedChannelException
at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source)
2021-08-16 17:30:52 ERROR TransportRequestHandler:293 - Error sending result RpcResponse{requestId=8861954111730219182, body=NioManagedBuffer{buf=java.nio.HeapByteBuffer[pos=0 lim=81 cap=156]}} to /192.168.56; closing connection
java.nio.channels.ClosedChannelException
at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source)
2021-08-16 17:30:52 ERROR TransportRequestHandler:293 - Error sending result RpcResponse{requestId=5535068542584258152, body=NioManagedBuffer{buf=java.nio.HeapByteBuffer[pos=0 lim=81 cap=156]}} to /192.168.562; closing connection
java.nio.channels.ClosedChannelException
at io.netty.channel.AbstractChannel$AbstractUnsafe.write(...)(Unknown Source)
2021-08-16 17:35:34 ERROR YarnClusterScheduler:70 - Lost executor 205 on compute006: Container container_e434_1628615141783_154721_01_000245 on host: compute006 was preempted.
2021-08-16 17:35:59 ERROR YarnClusterScheduler:70 - Lost executor 203 on compute007: Container container_e434_1628615141783_154721_01_000242 on host: compute007 was preempted.
2021-08-16 17:38:50 ERROR YarnClusterScheduler:70 - Lost executor 209 on data267: Container container_e434_1628615141783_154721_01_000241 on host: data267 was preempted.
2021-08-16 17:40:56 ERROR YarnClusterScheduler:70 - Lost executor 211 on data133: Container container_e434_1628615141783_154721_01_000248 on host: data133 was preempted.
2021-08-16 17:44:01 ERROR YarnClusterScheduler:70 - Lost executor 157 on data034: Container container_e434_1628615141783_154721_01_000185 on host: data034 was preempted.
2021-08-16 17:44:26 ERROR YarnClusterScheduler:70 - Lost executor 202 on data234: Container container_e434_1628615141783_154721_01_000244 on host: data234 was preempted.
2021-08-16 18:05:34 ERROR YarnClusterScheduler:70 - Lost executor 225 on data001: Container container_e434_1628615141783_154721_01_000262 on host: data001 was preempted.
2021-08-16 18:05:49 ERROR YarnClusterScheduler:70 - Lost executor 227 on data244: Container container_e434_1628615141783_154721_01_000264 on host: data244 was preempted.
2021-08-16 18:06:16 ERROR YarnClusterScheduler:70 - Lost executor 214 on data027: Container container_e434_1628615141783_154721_01_000251 on host: data027 was preempted.
2021-08-16 18:06:23 ERROR ApplicationMaster:43 - RECEIVED SIGNAL TERM
2021-08-16 18:06:23 ERROR ApplicationMaster:70 - User application exited with status 143
2021-08-16 18:06:23 ERROR FileFormatWriter:91 - Aborting job ea540d12-ad13-4e88-95fb-d8ac7f250503.
org.apache.spark.SparkException: Job 127 cancelled because SparkContext was shut down
The Spark app successfully runs more often than not with no errors. It sounds like 143 is typically OOM errors but my memory configs are pretty high:
'executor_memory': '10G',
'driver_memory': '12G',
'spark.executor.memoryOverhead': '5G',
'spark.driver.memoryOverhead': '4G',
What's the best way to get to the bottom of this?