How to access secrets in databricks initscript

Viewed 2316

I have tried to access the secret {{secrets/secrectScope/Key}} in advanced tab of databricks cluster and it is working fine. But when I try to use the same in databricks init script, it is not working it.

What are the steps to do that?

2 Answers

Here are the steps to access secrets in databricks initscript:

  1. Go to cluster
  2. Click Edit next to the Cluster information.
  3. On the Configure Cluster page, click Advanced Options.
  4. On the Spark tab, enter the following Spark Config:

enter image description here

Sample ini code:

 fs.azure.account.auth.type.chepragen2.dfs.core.windows.net OAuth
 fs.azure.account.oauth.provider.type.chepragen2.dfs.core.windows.net org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider
 fs.azure.account.oauth2.client.id.chepragen2.dfs.core.windows.net {<!-- -->{secrets/KeyVaultName/ClientID}}
 fs.azure.account.oauth2.client.secret.chepragen2.dfs.core.windows.net {<!-- -->{secrets/KeyVaultName/ClientSecret}}
 fs.azure.account.oauth2.client.endpoint.chepragen2.dfs.core.windows.net https://login.microsoftonline.com/<Directory_ID>/oauth2/token

For more details, refer Azure Databricks - configure the cluster to read secrets from the secret scope.

Related