Is it possible to change the output filenames when saving as partitioned parquet?

Viewed 123

When we save a pandas dataframe as a partitioned parquet, the filenames are generated automatically.

Is it possible to specify the output filenames of each partition?

Using an example

df = pd.DataFrame(data={'year':  [2020, 2020, 2021],
                        'month': [1,12,2], 
                        'day':   [1,31,28], 
                        'value': [1000,2000,3000]})

df.to_parquet('./output', partition_cols=['year', 'month'])


output/year=2020/month=1/6f0258e6c48a48dbb56cae0494adf659.parquet
output/year=2020/month=12/cf8a45116d8441668c3a397b816cd5f3.parquet
output/year=2021/month=2/7f9ba3f37cb9417a8689290d3f5f9e6e.parquet

Is it possible to get

output/year=2020/month=1/2020_01.parquet
output/year=2020/month=12/2020_12.parquet
output/year=2021/month=2/2021_02.parquet

Thanks for your time

0 Answers
Related