I’m trying to use org.apache.hadoop.fs.azure.NativeAzureFileSystem to read metadata of a file in ADSL2. I can't initialize the NativeAzureFileSystem.
This works:
Configuration conf = new Configuration();
conf.set("fs.azure.account.key.myaccount.blob.core.windows.net","<….. my ADLS2 storage accoune key>");
Path fsPath = new Path("wasbs://mycontainer@myaccount.blob.core.windows.net/");
NativeAzureFileSystem afs = new NativeAzureFileSystem();
afs.initialize(fsPath.toUri(),conf);
… but it’s actually working via Blob interface (wasbs). Here I’m trying to use abfss, and it doesn’t work:
Configuration conf = new Configuration();
conf.set("fs.azure.account.auth.type.myaccount.dfs.core.windows.net", "OAuth");
conf.set("fs.azure.account.oauth.provider.type.myaccount.dfs.core.windows.net", "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider");
conf.set("fs.azure.account.oauth2.client.id.myaccount.dfs.core.windows.net", "<.......>");
conf.set("fs.azure.account.oauth2.client.secret.myaccount.dfs.core.windows.net","<........>");
conf.set("fs.azure.account.oauth2.client.endpoint.myaccount.dfs.core.windows.net", "https://login.microsoftonline.com/<.....>/oauth2/token");
Path fsPath = new Path("abfss://mycontainer@myaccount.dfs.core.windows.net/");
NativeAzureFileSystem afs = new NativeAzureFileSystem();
afs.initialize(fsPath.toUri(),conf);
.. and I’m getting an error:
Caught: org.apache.hadoop.fs.azure.AzureException: org.apache.hadoop.fs.azure.AzureException: No credentials found for account myaccount.dfs.core.windows.net in the configuration, and its container delta is not accessible using anonymous credentials. Please check if the container exists first. If it is not publicly available, you have to provide account credentials.
org.apache.hadoop.fs.azure.AzureException: org.apache.hadoop.fs.azure.AzureException: No credentials found for account myaccount.dfs.core.windows.net in the configuration, and its container delta is not accessible using anonymous credentials. Please check if the container exists first. If it is not publicly available, you have to provide account credentials.
at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.createAzureStorageSession(AzureNativeFileSystemStore.java:1123)
at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.initialize(AzureNativeFileSystemStore.java:566)
at org.apache.hadoop.fs.azure.NativeAzureFileSystem.initialize(NativeAzureFileSystem.java:1423)
at org.apache.hadoop.fs.azure.NativeAzureFileSystem$initialize.call(Unknown Source)
I also tried setting up the conf without "myaccount.blob.core.windows.net" at the end of config entries, to no avail.
Am I missing some extra configuration here? The storage account is definitely enabled for hierarchical paths, and I can access it in Databricks using abfss.