ErrorCode=AzureDatabricksCommandError,Hit an error when running the command in Azure Databricks. Error details: com.databricks.backend.daemon.data.client.adl.AzureCredentialNotFoundException: Could not find ADLS Gen2 Token.
I'd appreciate any help in this :)
Found the solution, we have to use service principal to get through, azure AAD passthrough has lots of limitations.
We have to set the spark config as below:
fs.azure.account.oauth2.client.secret <<client secret>>
fs.azure.account.oauth2.client.endpoint https://login.microsoftonline.com/<<tenant id>>/oauth2/token
fs.azure.account.oauth2.client.id <<client id>>
fs.azure.account.key.<<datalake name>>.blob.core.windows.net <<data lake key>>
fs.azure.account.oauth.provider.type org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider
fs.azure.account.auth.type OAuth
spark.databricks.delta.preview.enabled true
Your data bricks workspace must have appropriate blob contributor (read/write) roles