Overwriting(Updating) Data on S3 via PySpark

user_14317174 0
Viewed 372

I have a pyspark script that will repeatedly be executed by airflow.

On each iteration, my script will load a dataframe from s3 location and update it.

However, when I try to update, it using

df.write.mode('overwrite').parquet(s3_url)

It completely wipes out the directory (deletes the directory). Am I misunderstanding the 'overwrite' mode ?

To put in short, I want to keep on overwriting to a certain s3 path without creating subfolders based on when I choose to run execute the script. When I tried to this with the code above, it deleted the existing directory.

pyspark apache-spark amazon-s3
1 Answers

The data is wiped out, before you start writing, so you don't have a data source. Using a data source, which is also a target, with overwrite mode, will require intermediate storage.

One (not ideal) way to solve this:

df.cache()
df.show()

df.write.mode('overwrite').parquet(s3_url)

This caches the data prior to writing, so the data source won't be needed. Note: Failures in this job, may still cause your target to be empty, but if it succeeds, the data should be in the target.

user_26281370
Related