Provide blob type to read an Azure append blob from PySpark

The ultimate goal is to able to read the data in my Azure container into a PySpark dataframe.

Steps until now

The steps I have followed till now:

• Written this code

  spark = SparkSession(SparkContext())
  spark.conf.set(
      "fs.azure.account.key.%s.blob.core.windows.net" % AZURE_ACCOUNT_NAME,
      AZURE_ACCOUNT_KEY
  )
  spark.conf.set(
      "fs.wasbs.impl",
      "org.apache.hadoop.fs.azure.NativeAzureFileSystem"
  )
  
  container_path = "wasbs://%s@%s.blob.core.windows.net" % (
      AZURE_CONTAINER_NAME, AZURE_ACCOUNT_NAME
  )
  blob_folder = "%s/%s" % (container_path, AZURE_BLOB_NAME)
  df = spark.read.format("text").load(blob_folder)
  print(df.count())
  

  • Set public access and anonymous access to my Azure container.
  • Added two jars hadoop-azure-2.7.3.jar and azure-storage-2.2.0.jar to the path.

Problem

But now I am stuck with this error: Caused by: com.microsoft.azure.storage.StorageException: Incorrect Blob type, please use the correct Blob type to access a blob on the server. Expected BLOCK_BLOB, actual UNSPECIFIED.. I have not been able to find anything which talks about / resolves this issue. The closest I have found is this which does not work / is outdated.

EDIT

I found that the azure-storage-2.2.0.jar did not support APPEND_BLOB. I upgraded to azure-storage-4.0.0.jar and it changed the error from Expected BLOCK_BLOB, actual UNSPECIFIED. to Expected BLOCK_BLOB, actual APPEND_BLOB.. Does anyone know how to pass the correct type to expect?

Can someone please help me with resolving this.

I have minimal expertise in working with Azure but I don't think it should be this difficult to read and create a Spark dataframe from it. What am I doing wrong?