I realize a Databricks cluster has a timeout, meaning after N minutes it will turn the cluster off. Here's a sample.
As nice as this feature is, though, it is not what we need. Our team works from 8AM to 6PM on weekdays. We want the cluster to would auto-start at 8AM, stay "always on" during working hours, THEN timeout after, say, 6PM. Make sense?
Q: Is this possible?
You can do everything inside the Databricks by scheduling some small job on the existing cluster. In this case, if cluster is stopped, then it will be started for execution of the job, and will stay until the auto-termination feature will kick-in (I would recommend to use 65-70 minutes as auto-termination setting to balance costs). You can create a notebook with something like
display(spark.range(1))
and schedule it for execution on selected cluster. To keep cluster running during the work hours, you need schedule the job to run periodically. This could be done with following cron expression (see Quartz docs for reference):
* 0 8-17 * * MON-FRI
P.S. Really, it should be maybe * 55 7-16 * * MON-FRI, to start 5 minutes before 8AM.
Yes, it possible to start the databricks cluster as per your team works from 8AM to 6PM on weekdays using Azure Automation.
To start at 8 AM you can use PowerShell runbook in Azure Automation to start your cluster as per the scheduled time as shown below:
PowerShell runbook should be as shown below:
$accessToken = "<Personal_Access_Token>"
$apiUrl = "<Azure_Databricks_Endpoint_URL>"
Set-DatabricksEnvironment -AccessToken $accessToken -ApiRootUrl $apiUrl
Start-DatabricksCluster -ClusterID "<Cluster_ID>"
To stop at 6 PM you can set the property Terminate after 600 minutes of inactivity.
Note: If your Business Hours (8AM to 6PM which means 10 Hours x 60 minutes) you can set the property Terminate after 600 minutes of inactivity as shown below:
This Tutorial: Start Azure Databricks clusters during business hours walks you through the creation of a PowerShell Workflow runbook to start Azure Databricks clusters during business hours in Azure Automation.
Unfortunately, It is not possible.
However, You can opt out of auto termination by clearing the Auto Termination checkbox or by specifying an inactivity period of 0.
Refer official document