Databricks job not failing after Exception

user_15025809 0
Viewed 413

I have a Databricks job which runs my Python notebook. I am running Databricks 7.4 lts. This job is triggered via Powershell.

If something goes wrong during the pipeline I raise an Exception. This exception is not caught. However, the job does not show as "Failed" after the exception is raised. Instead, the job shows "Successful". I can see the exception in the logs for the job.

Why is the job not showing as failed?

if pipeline_failure:
    raise Exception("Job should show failed")
databricks python
1 Answers

In the Databricks Python notebook I was calling display(dataframe) and raise Exception() in the same cell.

Specifically, display() was right before Exception()

After moving display(dataframe) out of the same cell as the Exception() statement the Databricks job to show "Failed" when the exception was raised.

user_150258090
Related