how to call spark.stop() old session?

user_8923003 0
Viewed 664

In Jupyter, I created spark session like this

spark = SparkSession.builder
         .master("yarn")
         .appName("test procjet")
         .getOrCreate()

But I forgot spark.stop() and shutdown Jupyter notebook. I want to call stop() to old session so I tried getOrCreate() but spark history server show new application.

How can I get old spark session and stop? I also tried yarn application -kill <appid> but spark history still show that.

jupyter-notebook apache-spark
1 Answers

When a Jupyter Notebook is shutdown, sparkSession used in that is also stopped as Jupyter is the client which created it. The same is the case when you forget to close() SparkSession in a script which uses it.

user_21429940
Related