How to read parquet files in pyspark from s3 bucket whose path is partially unpredictable?

user_1280726 0
Viewed 1036

My paths are of the format s3://my_bucket/timestamp=yyyy-mm-dd HH:MM:SS/.

E.g. s3://my-bucket/timestamp=2021-12-12 12:19:27/, however MM:SS part are not predictable, and I am interested in reading the data for a given hour. I tried the following:

  1. df = spark.read.parquet("s3://my-bucket/timestamp=2021-12-12 12:*:*/")
  2. df = spark.read.parquet("s3://my-bucket/timestamp=2021-12-12 12:[00,01-59]:[00,01-59]/")

but they give the error pyspark.sql.utils.IllegalArgumentException: java.net.URISyntaxException.

parquet apache-spark wildcard amazon-s3
2 Answers
user_8765950

If you want performance.....

I humbly suggest that when you do re-architect this you don't use S3 file lists/directory lists to accomplish this. I suggest you use a Hive table partitioned by hour. (Or you write a job to help migrate data into hours in larger files not small files.)

S3 is a wonderful engine for long term cheap storage. It's not performant, and it is particularly bad at directory listing due to how they implemented it. (And performance only gets worse if there are multiple small files in the directories).

To get some real performance from your job you should use a hive table (Partitioned so the file lookups are done in DynamoDB, and the partition is at the hour level.) or some other groomed file structure that reduces file count/directories listings required.

You will see a large performance boost if you can restructure your data into bigger files without use of file lists.

user_135351200
Related