How to kill apache spark application running in background after killing it from SPARK web UI

user_3535140 0
Viewed 2948

Code below is successfully created spark context when I submit using spark submit and running fine.

When I kill application under Running Applications from Apache spark web UI, application state shows killed but, printing Test application on screen after killing also:

Application running on apache spark web UI:

enter image description here

Application killed using "kill" button on spark web UI

enter image description here

Still printing message on screen after killing application

enter image description here

Need solution to auto kill python job when I kill spark-context

from pyspark import SparkConf
from pyspark import SparkContext

if __name__ == "__main__":
    conf = SparkConf().setAppName("TEST")
    conf.set("spark.scheduler.mode", "FAIR")
    sc = SparkContext(conf=conf)

    while True:
        print("Test application")
apache-spark sigkill python
3 Answers

You can do it the old fashioned way.

Run ps -ef and find the java job id. Then run kill -9 

//Find all the java jobs
[stack_overflow@stack_overflow ~]$ ps -ef | grep SparkSubmit
stack_overflow  96747  96736 99 11:19 pts/15   00:01:55 /usr/bin/java -cp /opt/spark/conf/:/opt/spark/jars/* -Dscala.usejavacp=true -Xmx1g -Dderby.system.home=/home/stack_overflow/Spark/ org.apache.spark.deploy.SparkSubmit --conf spark.local.dir=/opt/spark/temp_land/spark-temp --conf spark.driver.extraJavaOptions=-Dderby.system.home=/home/stack_overflow/ --class org.apache.spark.repl.Main --name Spark shell spark-shell
stack_overflow  97410  14952  0 11:20 pts/15   00:00:00 grep --color=auto SparkSubmit
//96747 is the Spark job I forced to become unresponsive
//97410 is the Base Spark Account don't delete
////Run the kill command on the job, only works if you have permissions on that job
[stack_overflow@stack_overflow ~]$ kill -9 96747
//The job is now dead and gone
[stack_overflow@stack_overflow ~]$ ps -ef | grep SparkSubmit
stack_overflow  96190  14952  0 11:17 pts/15   00:00:00 grep --color=auto SparkSubmit
user_98315220

You can open another session and see if your spark application is still running - 

yarn application -list <app_id>

then kill your application if still running,

yarn application -kill <app_id>
user_25603270

I found a way to solve my issue with below code. Thanks for all your responses

from pyspark import SparkConf
from pyspark import SparkContext

if __name__ == "__main__":
    conf = SparkConf().setAppName("TEST")
    conf.set("spark.scheduler.mode", "FAIR")
    sc = SparkContext(conf=conf)

    while True:
        if sc._jsc.sc().isStopped():
            break
        print("Test application")
user_35351400
Related