Listing files on Microsoft Azure Databricks

user_13798324 0
Viewed 658

I'm working in the Microsoft Azure Databricks. And using the ls command, I found out that there is a CSV file present in it (see first screenshot). But when I was trying to pick the CSV file into a list using glob, it's is returning an empty list (see second screenshot).

How can I list the contents of a directory in Databricks?

enter image description here

enter image description here

%fs 
    ls /FileStore/tables/26AS_report/normalised_consol_file_record_level/part1/customer_pan=AAACD3312M/

path = "/FileStore/tables/26AS_report/normalised_consol_file_record_level/part1/customer_pan=AAACD3312M/"
result = glob.glob(path+'/**/*.csv', recursive=True)
print(result)
pyspark databricks glob python-3.x python
2 Answers

glob is a local file-level operation that doesn't know about DBFS. If you want to use it, then you need to prepend a /dbfs to your path:

path = "/dbfs/FileStore/tables/26AS_report/....."
Alex_Ott_186270

I don't think you can use standard Python file system functions from the os.path or glob modules.

Instead, you should use the Databricks file system utility (dbutils.fs). See documentation.

Given your example code, you should do something like:

dbutils.fs.ls(path)

or

dbutils.fs.ls('dbfs:' + path)

This should give a list of files that you may have to filter yourself to only get the *.csv files.

user_106698750
Related