Call Databricks API from DevOps Pipeline using Service principal

user_6156353 0
Viewed 782

I want to be able to call Databricks API from DevOps pipeline. I can do this usint personal access token for my account, however I want to make API calls user independent so I wanted to use Service principal (App registration). I followed this tutorial https://docs.microsoft.com/en-us/azure/databricks/dev-tools/api/latest/aad/service-prin-aad-token to create access token for the service principal, however I have 2 issues:

  1. such generated token expires in 1 hour - is there any elegant was to automatically refresh it?
  2. even when calling the ADB API using this token I get 403 unauthorized - is there anything else I should do? The app registration has Contributor role for the ADB service.

EDIT: Added API Permission for the AzureDatabricks in App registration and Granted admin consent, however still no luck.

azure-pipelines azure-databricks databricks azure-devops azure
2 Answers
  1. such generated token expires in 1 hour - is there any elegant was to automatically refresh it?

No, client credentials flow doesn't support refresh token. You can try to get a new token, please refer to this issue.

  1. even when calling the ADB API using this token I get 403 unauthorized - is there anything else I should do? The app registration has User role for the ADB service.

Make sure your service principal have a Contributor role assigned.

There are two kinds of resource in different situations.

  1. API access for service principals that are Azure Databricks workspace users and admins

    resource=2ff814a6-3304-4ab8-85cb-cd0e6f879c1d

  2. API access for service principals that are not workspace users

    resource=https://management.core.windows.net/

user_133083810

So I found 3 possible solutions at the end.

  1. Generate access token for service principal, generate management service token for service principal and use both of these to access Databricks API - reference
  2. Use access token and management token to generate Databricks Personal access token for the service principal using Databricks Token API, then you can use it for Databricks CLI - reference
  3. Authenticate to Databricks via CLI using AAD token (reference and Databricks CLI help):
    1. az login --service-principal -u <app-id> -p <app-password> --tenant <tenant-id>
    2. token_response=$(az account get-access-token --resource 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d)
    3. export DATABRICKS_AAD_TOKEN=$(jq .accessToken -r <<< "$token_response")
    4. databricks configure --host https://<adb-url> --aad-token
user_61563530
Related