Python Pandas export to parquet, how to overwrite folder outputs

Viewed 504

I am pandas to read a csv file then export it to Parquet partitioned by date, it works great

import pandas as pd
import datetime
df = pd.read_csv('box.csv',parse_dates=True)
df['SETTLEMENTDATE'] = pd.to_datetime(df['SETTLEMENTDATE'])
df['Date'] = df['SETTLEMENTDATE'].dt.date
df.to_parquet('nem.parquet',partition_cols=['Date'],allow_truncated_timestamps=True)

and here is a sample output enter image description here

When I run the script again, it will generate a new files enter image description here

is there an Option in Pandas to overwrite the file instead, and add a new file only when there is a new data ?

0 Answers
Related