Databricks dbutils not displaying folder list under specfic folder

user_3192775 0
Viewed 980

I have three folders under a container

Structure of the folders

 folder1
   |_ file1.json
   |_ file2.json
 folder2
   |_ sub-folder1
       |_ file1.json
   |_ sub_folder2
       |_ sub-folder01
       |_ file2.json
 folder3
    |_ sub-folder1
        |_ file1.json

Note: folder2 only has list of folders, there might be files within the, i am trying to iterate and find specific file name in python code.

from pyspark.sql.functions import col,lit
from datetime import datetime

app_storage_acct_name= 'mystorageaccnt1'
app_storage_acct_scope="{}-scope".format(app_storage_acct_name)

config_secret_set_url = "fs.azure.account.key.{}.blob.core.windows.net".format(app_storage_acct_name)
secret = dbutils.secrets.get(scope = app_storage_acct_scope, key = app_storage_acct_key)
dbutils.fs.mount(
  source = "wasbs://mycontainer1@mystirageaccnt1.blob.core.windows.net",
  mount_point = "/mnt/my-data-src",
  extra_configs = {config_secret_set_url:dbutils.secrets.get(scope = app_storage_acct_scope, key = app_storage_acct_key)})

dbutils.fs.ls('/mnt/my-data-src/')

Above code prints three folder which i also see in the blob store explorer

Out[29]: [FileInfo(path='dbfs:/mnt/my-data-src/folder1/', name='folder1/', size=0),
 FileInfo(path='dbfs:/mnt/my-data-src/folder2/', name='folder2/', size=0),
 FileInfo(path='dbfs:/mnt/my-data-src/folder3/', name='folder3/', size=0)]

when i use below, the files are listed

dbutils.fs.ls('/mnt/my-data-src/folder1/')
  • output was like below
Out[30]: [FileInfo(path='dbfs:/mnt/my-data-src/folder1/file1.json', name='file1.json', size=1011),
 FileInfo(path='dbfs:/mnt/my-data-src....,

when i try to list the folders under the folder2 using

dbutils.fs.ls('/mnt/my-data-src/folder2/')
  • output java.io.FileNotFoundException: File /folder2 does not exist.
ExecutionError                            Traceback (most recent call last)
<command-2660727172978602> in <module>
----> 1 dbutils.fs.ls('/mnt/my-data-src/folder2/')

/databricks/python_shell/dbruntime/dbutils.py in f_with_exception_handling(*args, **kwargs)
    317                     exc.__context__ = None
    318                     exc.__cause__ = None
--> 319                     raise exc
    320 
    321             return f_with_exception_handling

ExecutionError: An error occurred while calling z:com.databricks.backend.daemon.dbutils.FSUtils.ls.
: java.io.FileNotFoundException: File /folder2 does not exist.
    at shaded.databricks.org.apache.hadoop.fs.azure.NativeAzureFileSystem.listStatus(NativeAzureFileSystem.java:2468)
    at com.databricks.backend.daemon.data.client.DBFSV2.$anonfun$listStatus$2(DatabricksFileSystemV2.scala:95)
    at com.databricks.s3a.S3AExceptionUtils$.convertAWSExceptionToJavaIOException(DatabricksStreamUtils.scala:66)
    at com.databricks.backend.daemon.data.client.DBFSV2.$anonfun$listStatus$1(DatabricksFileSystemV2.scala:92)
    at com.databricks.logging.UsageLogging.$anonfun$recordOperation$1(UsageLogging.scala:395)
    at com.databricks.logging.UsageLogging.executeThunkAndCaptureResultTags$1(UsageLogging.scala:484)
    at com.databricks.logging.UsageLogging.$anonfun$recordOperationWithResultTags$4(UsageLogging.scala:504)
    at com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:266)
    at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
    at com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:261)
    at com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:258)
    at com.databricks.backend.daemon.data.client.DatabricksFileSystemV2.withAttributionContext(DatabricksFileSystemV2.scala:510)
    at com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:305)
    at com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:297)
    at com.databricks.backend.daemon.data.client.DatabricksFileSystemV2.withAttributionTags(DatabricksFileSystemV2.scala:510)
    at com.databricks.logging.UsageLogging.recordOperationWithResultTags(UsageLogging.scala:479)
    at com.databricks.logging.UsageLogging.recordOperationWithResultTags$(UsageLogging.scala:404)
    at com.databricks.backend.daemon.data.client.DatabricksFileSystemV2.recordOperationWithResultTags(DatabricksFileSystemV2.scala:510)
    at com.databricks.logging.UsageLogging.recordOperation(UsageLogging.scala:395)
    at com.databricks.logging.UsageLogging.recordOperation$(UsageLogging.scala:367)
    at com.databricks.backend.daemon.data.client.DatabricksFileSystemV2.recordOperation(DatabricksFileSystemV2.scala:510)
    at com.databricks.backend.daemon.data.client.DBFSV2.listStatus(DatabricksFileSystemV2.scala:92)
    at com.databricks.backend.daemon.data.client.DatabricksFileSystem.listStatus(DatabricksFileSystem.scala:150)
    at com.databricks.backend.daemon.dbutils.FSUtils$.$anonfun$ls$1(DBUtilsCore.scala:154)
    at com.databricks.backend.daemon.dbutils.FSUtils$.withFsSafetyCheck(DBUtilsCore.scala:91)
    at com.databricks.backend.daemon.dbutils.FSUtils$.ls(DBUtilsCore.scala:153)
    at com.databricks.backend.daemon.dbutils.FSUtils.ls(DBUtilsCore.scala)
    at sun.reflect.GeneratedMethodAccessor223.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:380)
    at py4j.Gateway.invoke(Gateway.java:295)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:251)
    at java.lang.Thread.run(Thread.java:748)

Any specific reason why dbutils.fs.ls() not listing the folders which has folders in this case?

Answer: I tried to directly access a file and noticed since it was of blob type Append Blob. dbutils.fs.ls('/mnt/my-data-src/folder2/file.json) reports below message.

shaded.databricks.org.apache.hadoop.fs.azure.AzureException: hadoop_azure_shaded.com.microsoft.azure.storage.StorageException: Incorrect Blob type, please use the correct Blob type to access a blob on the server. Expected BLOCK_BLOB, actual APPEND_BLOB.

Is there any way to list the blob type append in databricks?

azure-databricks databricks python
2 Answers

Azure Databricks does support accessing append blobs using the Hadoop API, but only when appending to a file.

There is no workaround for this issue.

Use Azure CLI or Azure Storage SDK for Python to identify if the directory contains append blobs or the object is an append blob.

You can implement either a Spark SQL UDF or custom function using RDD API to load, read, or convert blobs using Azure Storage SDK for Python.

There is an official documentation given for this issue.

user_111048050

With little research, found the link in documentation. Official Doc

  • Finally there is a way to list those as files within the Databricks notebook. Refer the git sample link

Step 1. Install the azure-storage-blob module, with the temp cluster within the workspace.

%pip install azure-storage-blob

Step 2. Get the connection string of azure storeage and

from azure.storage.blob import ContainerClient

CONNECTION_STRING_OF_AZURE_BLOB_STORAGE='<connection-string-blob-storage-of-Access (IAM)>'

container = ContainerClient.from_connection_string(CONNECTION_STRING_OF_AZURE_BLOB_STORAGE, container_name="my-app-container")
#print(len(item))
blob_list = container.list_blobs()
for blob in blob_list:
   print(blob.name + '\n')
  • With the above code, I was able to list all the files in each folder.
user_31927750
Related