Is there an easy way to handle inconsistent file paths in blob storage?

user_14910109 0
Viewed 35

I have a service that drops a bunch of .gz files to an azure container on a daily cadence. I'm looking to pick these files up and convert the underlying txt/json into tables. The issue perplexing me is that the service adds two random string prefix folders and a date folder to the path.

Here's an example file path:

container/service-exports/z633dbc1-3934-4cc3-ad29-e82c6e74f070/2022-07-12/42625mc4-47r6-4bgc-ac72-11092822dd81-9657628860/*.gz

I've thought of 3 possible solutions:

  1. I don't necessarily need the data to persist. I could theoretically loop through each folder and look for .gz, open and write them to an output file and then go back through and delete the folders in the path.

  2. Create some sort of checkpoint file that keeps track of each path per gzip and then configure some way of comparison to the checkpoint file at runtime. Not sure how efficient this would be over time.

  3. Use RegEx to look for random strings matching the pattern/length of the prefixes and then look for the current date folder. If the date isn't today, pass.

Am I missing a prebuilt library or function capable of simplifying this? I searched around but couldn't find any discussions on this type of problem.

azure-blob-storage databricks python
1 Answers

You can do this using koalas.

import databricks.koalas as ks
path = "wasbs://container/service-exports/*/*/*.gz" 
df = ks.read_csv(path, sep="','", header='infer')

This should work out well if all the .gz files have the same columns, then df would contain all the data from .gz files concatenated.

user_148844540
Related