List all file names located in an Azure Blob Storage

user_19646212 0
Viewed 161

I want to list in Databricks all file names located in an Azure Blob Storage.
My Azure Blob Storage is structured like this:

aaa
<br/>------bbb
<br/>------------bbb1.xml
<br/>------------bbb2.xml
<br/>------ccc
<br/>------------ccc1.xml
<br/>------------ccc2.xml
<br/>------------ccc3.xml

If I do:

dbutils.fs.ls('wasbs://xxx@xxx.blob.core.windows.net/aaa')

only subfolders bbb and ccc are listed like this:

[FileInfo(path='wasbs://xxx@xxx.blob.core.windows.net/aaa/bbb/', name='bbb/', size=0), 
 FileInfo(path='wasbs://xxx@xxx.blob.core.windows.net/aaa/ccc/', name='ccc/', size=0)]

I want to deepen to the last subfolder to see all file names located in aaa: bbb1.xml, bbb2.xml, ccc1.xml, ccc2.xml and ccc3.xml.

If I do:

dbutils.fs.ls('wasbs://xxx@xxx.blob.core.windows.net/aaa/*')

an error occurs because the path can not be parameterized.

Any idea to do this in Databricks?

azure-blob-storage azure-databricks databricks azure
1 Answers

dbutils.fs.ls doesn't support wildcards, that's why you're getting an error. You have few choices:

  1. Use Python SDK for Azure blob storage to list files - it could be faster than using recursive dbutils.fs.ls, but you will need to setup authentication, etc.

  2. You can do recursive calls to dbutils.fs.ls, using function like this, but it's not very performant:

def list_files(path, max_level = 1, cur_level=0):
  """
  Lists files under the given path, recursing up to the max_level
  """
  d = dbutils.fs.ls(path)
  for i in d:
    if i.name.endswith("/") and i.size == 0 and cur_level < (max_level - 1):
      yield from list_files(i.path, max_level, cur_level+1)
    else:
      yield i.path
  1. You can use Hadoop API to access files in your container, similar to this answer.
Alex_Ott_186270
Related