Installing a python wheel on Databricks using own repository for wheel dependencies

user_9947884 0
Viewed 738

I am building a spark application that should run on Databricks. The application gets packaged into a wheel, pushed to the Databricks workspace and installed on a cluster. Due to my company's policies it is not possible to access PyPI directly but we must use an own internal, intermediate package repository. So trying to install the wheel fails as the application dependencies cannot be downloaded (obviously). The way of handling that in a regular pip install is to add the --index-url flag with the address of the intermediate repository: pip install my_wheel --index-url address/to/my/repo. In Databricks I am not pip installing a wheel but I am attaching it to a job (in the job definition) or an all purpose cluster so this is not possible.

databricks apache-spark
1 Answers

The way I solved it is by adjusting the environment variable PIP_INDEX_URL of the cluster to point to the internal repository. In the UI this can be done by setting PIP_INDEX_URL=address/to/my/repo at Cluster -> Edit -> Advanced Options -> Environment Variables. Using the API one can use spark_env_vars in the JSON defining the cluster.

{
  "name": "Test",
  "new_cluster": {
    "spark_version": "7.5.x-scala2.12",
    "node_type_id": "Standard_DS3_v2",
    "num_workers": 1,
    "spark_env_vars": {
      "PIP_INDEX_URL": address/to/my/repo
    }
  },
  "libraries": [
    {
      "whl": "dbfs:/my/wheel/path"
    }
  ],
  }
}
user_99478840
Related