I have some data on s3 in csv format. I read this data using pandas read_csv function. Now I have pandas dataframe. I want to save this dataframe on s3 in partitioned json format. for example
year=2020/
month=1/
day=1/
part-****.json
Also, it should Idempotence. Means I run the same program again it will overwrite existing partition and save the data.
I can easily do this in spark
df
.write
.mode("overwrite")
.partitionBy("year", "month", "day")
.json("s3://foo")
But I don't want to add pyspark dependency just to save data. Do we have any library available in python which can do this task??