How to create Azure databricks cluster using Service Principal

user_15048142 0
Viewed 312

I have azure databricks workspace and I added service principal in that workspace using databricks cli. I have been trying to create cluster using service principal and not able to figure it. Can any help me?

I am able to create cluster using my account but I want to create using Service Principal and want it to be the owner of the cluster not me. Also, it there a way I can transfer the ownership of my cluster to Service Principal?

azure-databricks azure
1 Answers

First, answering the second question - no, you can't change the owner of the cluster.

To create a cluster that will have Service Principal as owner you need to execute creation operation under its identity. To do this you need to perform following steps:

  • Prepare a JSON file with cluster definition as described in the documentation
  • Set DATABRICKS_HOST environment variable to an address of your workspace:
export DATABRICKS_HOST=https://adb-....azuredatabricks.net

  • Generate AAD token for Service principal as described in documentation and assign its value to DATABRICKS_TOKEN or DATABRICKS_AAD_TOKEN environment variables (see docs).

  • Create Databricks cluster using databricks-cli providing name of JSON file with cluster specification (docs):

databricks clusters create --json-file create-cluster.json

P.S. Another approach (really recommended) is to use Databricks Terraform provider to script your Databricks infrastructure - it's used by significant number of Databricks customers, and much easier to use compared with command-line tools.

Alex_Ott_186270
Related