We are submitting spark job into kubernetes cluster using cluster mode and with some more memory configurations. My job is finishing in about 5 mins but my executor pods are still running after 30 - 40 mins. Due to this the new jobs are pending as the resources are still bound to running pods.
Below is spark submit command :
/spark-2.4.4-bin-hadoop2.7/bin/spark-submit --deploy-mode cluster --class com.Spark.MyMainClass --driver-memory 3g --driver-cores 1 --executor-memory 12g --executor-cores 3 --master k8s://https://masterhost:6443 --conf spark.kubernetes.namespace=default --conf spark.app.name=myapp1 --conf spark.executor.instances=3 --conf spark.kubernetes.driver.pod.name=myappdriver1 --conf spark.kubernetes.container.image=imagePath --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark --conf spark.kubernetes.driver.container.image=imagePath --conf spark.kubernetes.executor.container.image=imagePath local:///opt/spark/jars/MyApp.jar