Can't connect to ADLS Gen2 from Azure Data Factory

sixplus4iszen 0
Viewed 79

I create a linked server to Azure Data Lake Storage Gen2 and test connection WITH file path is working.

After that I try to create a pipeline but get an error that I can't connect to the same file I tested the connection with.

linked server test OK

pipeline connect to same path fail

Full error msg:

ADLS Gen2 operation failed for: Storage operation '' on container 'cseo' and path 'standa...' get failed with 'Operation returned an invalid status code 'Forbidden''. Possible root causes: (1). It's possible because the service principal or managed identity don't have enough permission to access the data. (2). Please check storage network setting whether public network access is disabled. If disabled, use Managed Virtual Network IR and create Private Endpoint to access. https://docs.microsoft.com/en-us/azure/data-factory/managed-virtual-network-private-endpoint. https://docs.microsoft.com/en-us/azure/data-factory/tutorial-copy-data-portal-private (3). It's possible because some IP address ranges of Azure Data Factory are not allowed by your Azure Storage firewall settings. Azure Data Factory IP ranges please refer https://docs.microsoft.com/en-us/azure/data-factory/azure-integration-runtime-ip-addresses.. Account: 'cseo...'. FileSystem: 'cseo'. Path: 'standar...'. ErrorCode: 'AuthorizationPermissionMismatch'. Message: 'This request is not authorized to perform this operation using this permission.'. RequestId: '0b94...'. TimeStamp: '... GMT'.. Operation returned an invalid status code 'Forbidden' Activity ID: 29....

According to another answer I should be getting at leat contributor access but that was denied by the owner of the Data Lake.

From Databricks the same account works and I can access the table.

azure-data-lake-gen2 azure-data-factory
1 Answers

Since your connection test is successful, you can probably skip step 1a and step 1b regarding permissions. Most likely, especially when in an enterprise setup, you'll have difficulties connecting from the data factory runtime to a private endpoint.

1a) Did you set the correct IAM permissions?

From the error message you are getting, you should grant the service principal you are using access on the storage account with hierarchical namespaces enabled (adls gen2) one of the following roles:

  • Storage Blob Data Contributor (Read, Write, Delete)
  • Storage Blob Data Reader (Read)

1b) Are you using adls gen2 access control lists?

Using RBAC on adls gen2 storage accounts is not really fine-grained, because you are basically setting this on all files in the storage account. What you could also do is work with access control lists, those could be on a specific folder/file but also recursively. The downside here is that you could have certain permissions on one file but not on another file in the same folder. Make sure your connection test is using the same file that you are using in your data factory pipeline.

2a) Is your adls gen2 storage account accessible by private endpoint only

Then you should integrate your data factory into a private network as well. Ensure that this network is peered to the network containing the storage account and that the storage account has it's hostname registered in your private DNS and there is no Azure Firewall or NSG blocking traffic from data factory to the storage account.

2b) Is your adls gen2 storage account accessible only by IP addresses allowed by the firewall on the storage account?

Then you should integrate your data factory in a network as well. Ensure traffic flows out of that network through an azure firewall or similar appliance and whitelist the outbound IP on the storage account.

Cloudkollektiv0
Related