I have a project in which I instanciate a Spark Context, where I don't specify addPyFile() (nor start the App via spark-submit where I could specify --py-files).
However, there are several steps which rely on code in a local module dependencies. Nonetheless, everything works as expected, both locally - .master('local[*]') - and on a cluster - .master('yarn').
I import the necessary local modules before instanciating the Spark Context. Is this the reason why I don't need to pass the local dependencies via addPyFile()?
If not, why does it work in my case and when would it stop to work?
Earlier, I used spark-submit with --py-files to start the job (and it did not work without passing the dependencies via --py-files). Is it possible that the code somehow remained on the nodes so that it's available now? I would assume that everything is cleaned up afterwards.
Disclaimer: I'm pretty new to pyspark, so it's totally possible that I miss something obvious here.