Is it possible to connect to Azure Storage Account with Private Ednpoint from Databricks?

user_9609117 0
Viewed 548

I'm trying to connect to the Azure DataLake 2/Storage Account which is enabled with private endpoint from my Databricks Notebook, however getting error :

Caused by: StorageException: This Request is not Authorized to Perform this operation.

I tried adding Databricks IP addresses(found in Network Interface/ VNET attached with Databricks, but not sure if that's the one) in Firewall IP List to whilelist the requests from on-prem. But getting the same error. Is there a way to establish the connection with such configuration?

azure-storage-account azure-blob-storage databricks azure networking
1 Answers

I created the same environment as you have done and tried to mount the storage account in Databricks notebook but got the same error “Unauthorized”.

enter image description here

Solution:

As we have created the storage account with private endpoint there is a firewall setup by default . So, to fix that we need to have the vnet and the subnets that the databrick is using set inside it as shown below.

enter image description here

So after that when I mount the storage account it successfully gets mounted.

enter image description here

But again to read the files from the folder you have to Manage ACL’s for the container and also the files .

enter image description here

After this you can get the below output:

enter image description here

Same can be done for Files as well by doing a right click on the file that needs to be read from Databricks notebook.

enter image description here

You can test it in Databricks after you have added access control to the file:

enter image description here

Note: I have Used Service Principal(i.e. ADLStest) to use ADLS gen2 storage account from Databricks . That Service Principal should have “Storage Blob Data Contributor ” role assigned to it . And the Manage ACL's should be done for that service principal only.

Reference:

Access Azure Data Lake Storage Gen2 using OAuth 2.0 with an Azure service principal

Mount an Azure Data Lake Storage Gen2 Account in Databricks (mssqltips.com)

user_159692990
Related