In databricks using python, dbutils.fs.mount gives java.lang.NullPointerException: authEndpoint trying to mount using abfss. wasbs works fine

user_15794132 0
Viewed 1015

When using db.fs.mount in databricks to connect to azure gen2 data lake a authEndpoint error is received when attempting to connect to "abfss://theDir@theDataLake.blob.core.windows.net/" HOWEVER, connecting to "wasbs://theDir@theDataLake.blob.core.windows.net/" works fine. I am trying to understand why abfss causes the authEndpoint error while wasbs does not.

enter code here
#fails
endpoint = "abfss://theDir@theDataLake.blob.core.windows.net/";
dbutils.fs.mount(
   source = endpoint,
   mount_point = "/mnt/test",
   extra_configs = {"fs.azure.account.key.theDataLake.blob.core.windows.net" : "xxxxxx"})

#works
endpoint = "wasbs://theDir@theDataLake.blob.core.windows.net/";
dbutils.fs.mount(
   source = endpoint,
   mount_point = "/mnt/test",
   extra_configs = {"fs.azure.account.key.theDataLake.blob.core.windows.net" : "xxxxxx"})
azure-data-lake-gen2 azure-databricks databricks python
1 Answers

You can't mount the ABFSS protocol using the storage key. You can mount with ABFSS only when using Service Principal (docs), and it requires another set of parameters for extra_configs:

{"fs.azure.account.auth.type": "OAuth",
"fs.azure.account.oauth.provider.type": "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider",
"fs.azure.account.oauth2.client.id": "<application-id>",
"fs.azure.account.oauth2.client.secret": dbutils.secrets.get(scope="<scope-name>",key="<service-credential-key-name>"),
"fs.azure.account.oauth2.client.endpoint": "https://login.microsoftonline.com/<directory-id>/oauth2/token"}
Alex_Ott_186270
Related