The scenario : we have pyspark jobs running using Kubernetes Spark Operator
The final step in each spark job is SparkContext stop()
Particularly with jobs taking longer than 40-50 minutes to complete, all the executor pods terminate gracefully at the end, however frequently the driver pod does not end. Interestingly no failures or errors showing up in the logs. The only difference observed from logs is that the shutdown hook does not get called.
What would be a clean way to ensure the driver pod terminates in such case ? Is sys.exit(0) needed in addition to the spark context stop()