Get Notified if hosted Spark Session dies

user_328968 0
Viewed 283

I have a long running Python 3.6 application that is hosting a PySpark 2.4.6 Session on YARN. I would like to be able to get notified/have a method called if the Spark Session crashes/dies, so that I can automatically restart it.

I'd like to do this proactively, rather than wrapping every call to the session and catching errors that the session is closed, and then making users wait while the session restarts.

pyspark apache-spark
2 Answers

There is a REST API available for Yarn. In intervals you could query all currently running applications

http://rm-http-address:port/ws/v1/cluster/apps?states=RUNNING

and then check if your PySpark session is part of the result.

user_21298010

Using https://stackoverflow.com/a/44084038/328968 as a starting point, you can create a listener and add it to the session. When the application ends you can perform a callback to restart the application.

SparkListener is defined in the above ref'd Answer.

class SparkApplicationEndListener(SparkListener):
    def __init__(self, applicationEndCallback):
        self.applicationEndCallback = applicationEndCallback

    def onApplicationEnd(self, applicationEnd):
        if self.applicationEndCallback != None:
            self.applicationEndCallback(applicationEnd)

########

def handleSparkApplicationEnd(app_end):
    print(app_end.toString())
    sparkSession = buildSparkSession()

def buildSparkSession():
    #......
    sparkSession.sparkContext._gateway.start_callback_server()
    listener = SparkApplicationEndListener(handleSparkApplicationEnd)
    sparkSession.sparkContext._jsc.sc().addSparkListener(listener)
user_3289680
Related