I would like to reuse the same Spark application across multiple runs of a python script, that uses it's Spark session object. That is, I would like to have a Spark application running in the background, and access it's Spark session object from within my script. Does anybody know how to do that?
1 Answers
To the best of my knowledge, it is not possible. It is the security model of Spark to isolate each session to a distinct app.
What I have done in the past:
build a small REST server on top of Spark that listens to specific command. At boot time, the server creates the session and load the data, so that forthcoming transformations are fast.
cache data in Delta lake, you still have the boot time and data ingestion, but it’s much faster that accessing data from several sources and preparing the data.
If you describe a bit more your use-case, I may be able to help a little more.
Related