Good Morning,
I am currently working on running and monitoring Databricks Notebooks from Airflow. I am currently able to login, spin up a new cluster, and execute a notebook. The issue I have is the monitoring side of things. Using the 2.1 jobs API I am making the following call.
curl --location --request GET 'https://<MY-SERVER>.azuredatabricks.net/api/2.1/jobs/runs/list?job_id=<MY-Job-Id>
On the API page it looks like that API should feed back some stats about the job itself. Most importantly to me is state.result_state as I want to use it as a sensor.
My issue is that when I hit the list endpoint for any of my jobs the only thing I get back is
{
"has_more": false
}
I cant find anywhere in the documentation where I need to add something in the Notebook itself to omit the metric im looking for. Is there maybe an elevated set of permissions I need for my Auth Token that would give me the full set of metrics I am looking for?