I am reading CSV files from datalake store, for that I am having multiple paths but if any one path does not exist it gives exception. I want to avoid this expection.
2 Answers
I think if you want to check for multiple pathes, the check will fail if one path does not exist. Perhaps you could try a different approach.
For the given example if you want to subselect subfolders you could try the following instead.
Read sub-directories of a given directory:
# list all subfolders and files in directory demo
dir = dbutils.fs.ls ("/mnt/adls2/demo")
Filter out the relevant sub-directories:
pathes = ''
for i in range (0, len(dir)):
subpath = dir[i].path
if '/corr' in subpath or '/deci' in subpath and subpath.startswith ('dbfs:/'): # select dirs to read
pathes = pathes + (dir[i].path) + ' '
# convert the string to a list
pathes = list(pathes.split())
Use the result-list to read the dataframe:
df = (spark.read
.json(pathes))
where path is a specific variable, you can use to check if it exists (scala):
dbutils.fs.ls("/mnt").map(_.name).contains(s"$path/")
Related