How to start and stop spark Context Manually

user_1080145 0
Viewed 40107

I am new to spark.In my current spark application script, I can send queries to spark in-memory saved table and getting the desired result using spark-submit.The problem is, each time spark context stops automatically after completing result. I want to send multiple queries sequentially.for that I need to keep alive spark context. how could I do that ? my point is

Manual start and stop sparkcontext by user

kindly suggest me.I am using pyspark 2.1.0.Thanks in advance

pyspark apache-spark
2 Answers

Try this code:

conf = SparkConf().setAppName("RatingsHistogram").setMaster("local")

sc = SparkContext.getOrCreate(conf)

This ensures to don;t have always stop your context and at the same time, if existing Spark Context are available, it will be reused.

user_24220700
Related