Issue with Dataproc Pyspark job not exiting even though logs have errors

Can see error in logs multiple times for my pyspark job in dataproc, but the job doesn't exit and keeps on running for multiple hours.

Any help to solve this is much appreciated.

The data on which the job is running is very small also.

Sometimes after rerun, the code job runs fine. But it picks up this issue randomly

Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
executor.scala:318)
                at org.apache.spark.executor.Executor.$anonfun$stopHookReference$1(Executor.scala:76)
                at org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:214)
                at org.apache.spark.util.SparkShutdownHookManager.$anonfun$runAll$2(ShutdownHookManager.scala:188)
                at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
                at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1996)
                at org.apache.spark.util.SparkShutdownHookManager.$anonfun$runAll$1(ShutdownHookManager.scala:188)
                at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
                at scala.util.Try$.apply(Try.scala:213)
                at org.apache.spark.util.SparkShutdownHookManager.runAll(ShutdownHookManager.scala:188)
                at org.apache.spark.util.SparkShutdownHookManager$$anon$2.run(ShutdownHookManager.scala:178)
                at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
                at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
                at java.lang.Thread.run(Thread.java:750)
WARN org.apache.spark.executor.Executor: Unable to stop heartbeater
java.lang.NullPointerException
                at org.apache.spark.executor.Executor.stop(Executor.scala:324)
                at org.apache.spark.executor.Executor.$anonfun$stopHookReference$1(Executor.scala:76)
                at org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:214)
                at org.apache.spark.util.SparkShutdownHookManager.$anonfun$runAll$2(ShutdownHookManager.scala:188)
                at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
                at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1996)
                at org.apache.spark.util.SparkShutdownHookManager.$anonfun$runAll$1(ShutdownHookManager.scala:188)
                at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
                at scala.util.Try$.apply(Try.scala:213)
                at org.apache.spark.util.SparkShutdownHookManager.runAll(ShutdownHookManager.scala:188)
                at org.apache.spark.util.SparkShutdownHookManager$$anon$2.run(ShutdownHookManager.scala:178)
                at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
                at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
                at java.lang.Thread.run(Thread.java:750)
ERROR org.apache.spark.util.Utils: Uncaught exception in thread shutdown-hook-0
java.lang.NullPointerException
                at org.apache.spark.executor.Executor.$anonfun$stop$3(Executor.scala:333)
                at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
                at org.apache.spark.util.Utils$.withContextClassLoader(Utils.scala:222)
                at org.apache.spark.executor.Executor.stop(Executor.scala:333)
                at org.apache.spark.executor.Executor.$anonfun$stopHookReference$1(Executor.scala:76)
                at org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:214)
                at org.apache.spark.util.SparkShutdownHookManager.$anonfun$runAll$2(ShutdownHookManager.scala:188)
                at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
                at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1996)
                at org.apache.spark.util.SparkShutdownHookManager.$anonfun$runAll$1(ShutdownHookManager.scala:188)
                at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
                at scala.util.Try$.apply(Try.scala:213)
                at org.apache.spark.util.SparkShutdownHookManager.runAll(ShutdownHookManager.scala:188)
                at org.apache.spark.util.SparkShutdownHookManager$$anon$2.run(ShutdownHookManager.scala:178)
                at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
                at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
                at java.lang.Thread.run(Thread.java:750)


.
WARN org.apache.spark.scheduler.cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: Requesting driver to remove executor 925 for reason Container from a bad node: container_1657869605389_0001_01_000925 on host: xxxx  Exit status: 1. Diagnostics: Exception from container-launch.
Container id: container_1657869605389_0001_01_000925
Exit code: 1