Reading/writing data in one azure gen2 storage account in multiple paths using multiple linked services in synapse notebook

user_8647357 0
Viewed 1074

Reading/writing data in one azure gen2 storage account in multiple paths using multiple linked services in synapse notebook. we can't read with single linked service as we have multiple SPN which has the access to all the paths.

Steps I followed to implement the same 1. Created a multiple linked service pointing to the gen2 storages using varies SPN1 , SPN2
linked_service_name = 'ln_spn1_path_1' spark.conf.set('spark.storage.synapse.linkedServiceName',linked_service_name) spark.conf.set('fs.azure.account.oauth.provider.type','com.microsoft.azure.synapse.tokenlibrary.LinkedServiceBasedTokenProvider')

  1. Created linked service ln_spn2_path_2 using spn 2 and pointing to the path2

    **linked_service_name = 'ln_spn2_path_2' spark.conf.set('spark.storage.synapse.linkedServiceName',linked_service_name) spark.conf.set('fs.azure.account.oauth.provider.type','com.microsoft.azure.synapse.tokenlibrary.LinkedServiceBasedTokenProvider')

     mssparkutils.fs.ls("abfss://container@storaegaccount.dfs.core.windows.net/path2")**

If above both the commonds run in same synapse notebook so we will get the below error which ever we run later

Py4JJavaError: An error occurred while calling z:mssparkutils.fs.ls. : java.nio.file.AccessDeniedException: Operation failed: "This request is not authorized to perform this operation using this permission.", 403, GET, AuthorizationPermissionMismatch, "This request is not authorized to perform this operation using this permission. RequestId:8853fd20-001f-0018-0347-bf3712000000 Time:2021-10-12T08:57:13.0513182Z"

Can we read the data using to different linked services in same notebook?

azure-synapse azure
1 Answers

UPDATE: I had tested on azure gen2 storage account in multiple paths using multiple linked services in synapse notebook.

Linked service: ADLSGen2_chepra with the notebook named Linked_service_name

enter image description here

Linked service: AzureDataLakeStorage_chepragen2 with the notebook named Linked_service_name

enter image description here

Note: Storage Blob Data Contributor: Use to grant read/write/delete permissions to Blob storage resources.

If you are not assigning "Storage Blob Data Contributor" to other synapse users, they will be not able to access the data from ADLS gen2 due to the lack of permission on the storage account.

enter image description here

If they try to access data from ADLS gen2 without the "Storage Blob Data Contributor" role on the storage account, they will receive the following error message:

Py4JJavaError: An error occurred while calling z:mssparkutils.fs.ls. : java.nio.file.AccessDeniedException: Operation failed: "This request is not authorized to perform this operation using this permission.", 403

enter image description here

Reference: Grant permissions to workspace managed identity.

user_81884330
Related