Reading data from blob without accountkey with pyspark

user_7565232 0
Viewed 659

I'm new to Azure and I would like to read a blob using pyspark but without using the account key. I used to read that blob using DefaultAzureCredential. I have tried generating a SAS Token and then running these 2 lines:

configuration = pyspark.SparkConf().set(
  "fs.azure.sas."+ container_name + ".blob.core.windows.net",
  sas)

spark = SparkSession \
    .builder \
    .config(conf = configuration) \
    .appName("Read csv file") \
    .getOrCreate()

df = spark.read.csv(location)

but I got this error :

org.apache.hadoop.fs.azure.AzureException: org.apache.hadoop.fs.azure.AzureException: No credentials found for account XXXXXXXXX in the configuration, and its container XXXXXXXX is not accessible using anonymous credentials. Please check if the container exists first. If it is not publicly available, you have to provide account credentials.
    at org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.createAzureStorageSession(AzureNativeFileSystemStore.java:1098)

To generate the sas I used the generate_blob_sas function. Does it mean there's no way to read the data without the account key?

pyspark azure python
1 Answers

We can read files from the blob using only SAS tokens, but in order to extract data from the blob, we must specify the correct path, storage account name, and container name.

This is how I was able to read the blob.

blob_account_name = "<Your Storage Account Name>"
blob_container_name = "<Your Container Name>"
blob_relative_path = "<Relative Path or Directly Mention your blob Name>"
blob_sas_token = "<Your Shared Access Signature Token>"

wasbs_path = 'wasbs://%s@%s.blob.core.windows.net/%s' % (blob_container_name, blob_account_name, blob_relative_path)
spark.conf.set('fs.azure.sas.%s.%s.blob.core.windows.net' % (blob_container_name, blob_account_name), blob_sas_token)

df = spark.read.csv(wasbs_path)

REFERENCES: Run a Spark job on Azure Databricks Workspace using Azure portal

user_159699810
Related