I have an Azure Data Factory pipeline that runs few Azure Databricks Notebooks every day. I keep having this problem that the notebook instance keeps running for a long time. When I checked, I see "Waiting for the cluster to start" in the cell output. But, when I checked the cluster, its in a running state. I thought this might be temporary and waited for another 30 minutes and it the cell output was still "Waiting for the cluster to start". Is there a way to force re-check cluster status inside the notebook? if Not, what's the correct way to handle this error
0 Answers
Related