How to determine if a Databricks cluster is ready using the API?

user_98215 0
Viewed 476

I'm calling the /clusters/events API with PowerShell to check if my Databricks cluster is up and ready for the next step in my setup process. Is this the best approach?

Currently, I grab the array of ClusterEvent and check the most recent ClusterEvent for its ClusterEventType. If it's RUNNING, we're good to go and we move on to the next step.

Recently, I discovered my release pipeline was hanging while checking the cluster status. It turns out that the cluster was in fact running but its status was DRIVER_HEALTHY, not RUNNNING. So, I changed my script and everyone is happy again.

Is there an official API call I make that returns yes/no, true/false, etc. so I don't need to code for the ClusterEventType I find means the cluster is running?

azure-databricks databricks
1 Answers

There is no such API that says yes/no about the cluster status. You can use Get command of the Clusters REST API - it returns information about current state of the cluster, so you just need to wait until it's get to the RUNNING state.

P.S. if you're doing that as part of release pipeline, or something like, then you can look to the Terraform provider for Databricks - it will handle waiting for cluster running, and other things automatically, and you can combine it with other things, like, provisioning of Azure resources, etc.

Alex_Ott_186270
Related