How to save pandas dataframe into partitioned JSON

Viewed 292

I have some data on s3 in csv format. I read this data using pandas read_csv function. Now I have pandas dataframe. I want to save this dataframe on s3 in partitioned json format. for example

year=2020/
    month=1/
        day=1/
            part-****.json

Also, it should Idempotence. Means I run the same program again it will overwrite existing partition and save the data.

I can easily do this in spark

df
  .write
  .mode("overwrite")
  .partitionBy("year", "month", "day")
  .json("s3://foo")    

     

But I don't want to add pyspark dependency just to save data. Do we have any library available in python which can do this task??

0 Answers
Related