Check for S3 directory existence in Databricks notebook

user_13014864 0
Viewed 853

I am working in Databricks and I am trying to grab parquet data from S3 instances that are ordered in a sequence, except some parts of the sequence are missing. So the file system might look like this:

 's3a://databricks-data/STAGING/18',
 's3a://databricks-data/STAGING/17',
 's3a://databricks-data/STAGING/16',
 's3a://databricks-data/STAGING/15',
 's3a://databricks-data/STAGING/14',
 's3a://databricks-data/STAGING/13',
 's3a://databricks-data/STAGING/12',
 's3a://databricks-data/STAGING/10',
 's3a://databricks-data/STAGING/09',
 's3a://databricks-data/STAGING/08',
 's3a://databricks-data/STAGING/07'

You'll notice that 11 is missing and this is where the problem comes in. There are occasionally some of these folders missing, and it's not systematic or predictable. So, coming from a Python background, I figured I could create a list of folders from a sequence and then test whether such a folder exists, and if so, read the data, if not, skip. Something like this:

paths = ["s3a://databricks-data/STAGING/" + str(ii) for ii in range(100)]
paths = [p for p in paths if p.exists()] #**this check -- "p.exists()" -- is what I'm looking for**
df = spark.read.parquet(*paths)

Does anyone know how I can check if a folder/directory exists in Databricks? I've tried using dbutils, but either I'm working with a different version or I don't know what I'm doing (maybe both), because nothing works. I'm pretty new to Databricks so let me know if I can clarify this question at all.

pyspark databricks amazon-s3
1 Answers

You can use dbutils.fs.ls for example to implement such function:

def path_accessible(path):
  try:
    dbutils.fs.ls(path)
    return True
  except:
    pass

  return False

and then use it:

paths = [p for p in paths if path_accessible(p)]

Please note, that it checks if the path is accessible - you may not have access because of invalid credentials, etc. - you can improve it by adding corresponding except clauses.

P.S. If you're doing that to load the increments of data, I would recommend to consider Structured Streaming (even in the Trigger.Once mode) with Auto Loader - it could be better from performance point of view if you have a lot of files in your directories.

Alex_Ott_186270
Related