Pyspark: How to check whether a file path with wild character exists in s3

user_11941902 0
Viewed 518

I want to check whether a file exists in an s3 path and then read it as a spark dataframe. The problem is I do not know the exact path of the file, so I have to use wild characters. For example, my path looks something like this:

path = 's3a://dir1/dir2/someHashString-2021-09-29-14-52-randomStuff.gz'

I have no way of knowing what 'someHashString' or 'randomStuff' is, so, I probably have to use wild characters like this

path = 's3a://dir1/dir2/*-2021-09-29-14-52-*'

I am trying the first solution mentioned here like this :

def path_exists(path):
    # spark is a SparkSession
    sc = spark.sparkContext
    fs = sc._jvm.org.apache.hadoop.fs.FileSystem.get(
         sc._jvm.java.net.URI.create("s3a://" + path.split("/")[2]),
         sc._jsc.hadoopConfiguration(),
                                                     )
    return fs.exists(sc._jvm.org.apache.hadoop.fs.Path(path))

path_exists(path='s3a://dir1/dir2/*-2021-09-29-14-52-*')

But it returns False even such a path does exist. How to check if this file path exists?

pyspark apache-spark amazon-web-services amazon-s3
1 Answers

FileSystem method exists does not support wildcards in the file path to check existence.

You can instead use globStatus which supports special pattern matching characters like *. If it returns a non-empty list then the file exists else it does not exist:

def path_exists(path):
    hpath = sc._jvm.org.apache.hadoop.fs.Path(path)
    fs = hpath.getFileSystem(sc._jsc.hadoopConfiguration())

    return len(fs.globStatus(hpath)) > 0


print(path_exists("s3a://dir1/dir2/*-2021-09-29-14-52-*"))
# True
user_13865510
Related