get all the contents of data lake gen2 folder in a list azure synapse workspace

user_7931463 0
Viewed 1153

I am brand new to Azure. I have created a data lake gen2 storage account and a container inside it and saved some files and folders in it.I want to list all the files and folders in azure synapse notebook so that i can process a particular file. I am using this command

mssparkutils.fs.ls("abfss://iogen2@demoadlsgen2.dfs.core.windows.net/first/")

but it giving me only one output like:

[FileInfo(path=abfss://iogen2@demoadlsgen2.dfs.core.windows.net/first/stocks, name=stocks, size=0]

I want my answer in a list like:

'abfss://iogen2@demoadlsgen2.dfs.core.windows.net/first/stocks/',
'abfss://iogen2@demoadlsgen2.dfs.core.windows.net/first/stocks/2022-03-06/',
'abfss://iogen2@demoadlsgen2.dfs.core.windows.net/first/stocks/2022-03-06/csv_files/',
'abfss://iogen2@demoadlsgen2.dfs.core.windows.net/first/stocks/2022-03-06/csv_files/demo.csv'

Apparently when i am using os.listdir it is giving an error:

FileNotFoundError: [Errno 2] No such file or directory:

Can anyone please help me in this

azure-data-lake-gen2 azure-synapse pandas azure list
1 Answers

As per the repro from my end, it shows all the files in the folder.

Here is files contained in the folder named sample:

enter image description here

I'm able to get the all the files contained in the folder named sample:

enter image description here

If you want to use the os.listdir you need to use file mount/unmount API in Synapse.

enter image description here

user_81884330
Related