I am trying to get all parameters related to a Databricks job and import them into python. These parameters should include the date, start time, duration, Status of the job(successful or failed) and all other parameters related to it.
I want to use these parameters to generate a daily report using python and the report has to contain details of all the jobs that ran either successfully or failed for everyday of the week.
You can use Databricks Jobs REST API for that - you can list all jobs, and for each jobs get all runs. The main problem is that you will scrape that API again & again.
Another possibility will be to get information about executed logs from the diagnostic logs on Azure or audit log on AWS - from it you can get extract information about which job run & what IDs were associated with them. Then you can get more detailed information using the get run API.
Or you can setup a tool called Overwatch in your workspace, and it will collect all of this information for you, plus much more information that is very useful for finding not optimal jobs, providing cost estimates, and a lot of other functionality.
