How to setup Databricks job using credentials passthrough on Azure

BI Dude 0
Viewed 36

I want to stop using mounts and tighten up security and reduce cost using job clusters.

I want to use passthrough credentials. I've created a notebook that reads avro and converts it to parquet. It works when I run the notebook; however, job fails when it runs as service principal or my account. Service principal is blob contributor. I'm not able to run working notebook as a job.

I've tried using job cluster and existing high concurrency cluster and I get error below in both cases:

Error message:
Py4JJavaError: An error occurred while calling o498.load.
: com.databricks.backend.daemon.data.client.adl.AzureCredentialNotFoundException: Could not find ADLS Gen2 Token
azure-active-directory azure-databricks databricks
1 Answers

AAD credentials passthrough doesn't work for jobs, especially for jobs owned by service principals. AAD passthrough relies on capturing the user's AAD token and forwarding it to ADLS...

But if you're already using the service principal, why not configure the job for direct access to ADLS as it's described in the documentation? You just need to set Spark conf properties in the job cluster (replace values in <> with actual values):

fs.azure.account.auth.type OAuth
fs.azure.account.oauth.provider.type org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider
fs.azure.account.oauth2.client.endpoint https://login.microsoftonline.com/<azure-tenant-id>/oauth2/token
fs.azure.account.oauth2.client.id <sp-client-id>
fs.azure.account.oauth2.client.secret {{secrets/<secret-scope>/<secret-key>}}
Alex Ott0
Related