spark.read.format("parquet") - Path does not exist

Question

I'm trying to load a list of parquets based on a range of days (GCP, PySpark), but I don't know if every item within the range exits, so I checked the existence first.

start_day = '20220701'
start_day = datetime.strptime(start_day, '%Y%m%d')
X = 30

dates = [start_day+timedelta(days=x) for x in range(X)]

table_path = "gs://my_folder/parquet/year={year}/month={month}/day={day}/*"

paths = [table_path.format(year=date.year, month=date.month, day=date.day) for date in dates]

for path in paths:
    try:
        spark.read.format("parquet").load(path)
    except:
        print(path)
        paths.remove(path)

df = spark.read.format("parquet").load(paths)

The code seems to works fine but not for every parquet. It deletes the days 1, 3 and 5 from the list:

gs://my_folder/parquet/year=2022/month=7/day=1/*
gs://my_folder/parquet/year=2022/month=7/day=3/*
gs://my_folder/parquet/year=2022/month=7/day=5/*

but fails when trying to load the parquet for day 2:

AnalysisException: Path does not exist: gs://my_folder/parquet/year=2022/month=7/day=2/*

I have checked it manually and days 2 and 6 should have been deleted too.

What am I missing?

Thank you in advance.

spark.read.format("parquet") - Path does not exist

0 Answers