I have a long-running job, and if certain conditions are met, I would like to kill the job. This is traditionally done in python like:
if some_condition:
exit('job failed!)
This works on traditional, command-line running of a python script. However, for reasons beyond my understanding, Databricks doesn't kill the script with exit.
How can I program a Databricks python script to automatically stop executing if a certain condition is met?
I think this is due to where in the overall stack the Python REPL exists, causes exit() to not actually exit the JVM that Spark is inside of. The exception you are looking for to exit a spark job is probably SparkException - https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SparkException.scala
Alternatively, in Databricks, you can use dbutils.notebook.exit() https://docs.databricks.com/user-guide/notebooks/notebook-workflows.html#notebook-workflows