How can we get the DataBricks Workspace URL and Access Token

Syamala Kumar Rama Koti Reddy 0
Viewed 38

I am trying to do the End to End implementation of Databricks in CI/CD integration for

Ex: Creation of Cluster , Install libraries and other things using Rest API

So for this we need Domain(Workspace URL) and Token(Access Token) of workspace

How can I get the workspace url and access token by automatically so I can use them in the REST API code using python.

For ex:

This the task for create cluster in release pipeline

def cluster_create():
    response = requests.post(
        'https://%s/api/2.0/clusters/create' % (DOMAIN),
        headers={'Authorization': 'Bearer %s' % TOKEN},
        json={
     "cluster_name": "cluster",
     "spark_version": "7.3.x-scala2.12",
    "node_type_id": "Standard_DS3_v2",
    "autotermination_minutes": 10,
    "autoscale" : {
      "min_workers": 1,
      "max_workers": 3
    }
  }
    )
    if response.status_code == 200:
        print(response.json()['cluster_id'])
        os.environ["DBRKS_CLUSTER_ID"] = response.json()["cluster_id"]
        return response.json()['cluster_id']

    else:
        print("Error launching cluster: %s: %s" %
              (response.json()["error_code"], response.json()["message"]))

Steps:

  1. In Build pipeline I am creating a data brick workspace with ARM template.
  2. This is the function I will be using in the release pipeline for creating cluster.

Once I created the Data Bricks workspace How to get Domain or Token in release pipeline to replace the variables in create cluster task automatically for continuous integration and continuous development

azure-pipelines azure-devops azure-databricks python
1 Answers

If the workspace is created in the same pipeline, and it's created via service principal or managed identity, then you can use Azure Active Directory tokens to authenticate to the workspace. The detailed instructions on the generation of the tokens are in the documentation. Please note that in your case, that service principal or managed identity isn't the part of workspace yet, so you need to follow instructions in section API access for service principals that are not workspace users where you need to generate two AAD tokens.

P.S. Really, I would recommend to use Terraform Databricks provider - it will make it simpler to create & update objects. We have public examples of CI/CD with terraform.

Original answer (before question was refined):

Standard method is to put this data into Azure DevOps variables (or variables group) and use from your pipeline.

P.S. I have an example of CI/CD pipeline for notebooks, you can find some instructions in the README.

Alex Ott0
Related