How to control file size in Pyspark?

user_15128981 0
Viewed 530

I'm writing a dataframe to csv with the following code:

df.write\
    .option("header",True) \
    .mode("overwrite") \
    .option("sep","|")\
    .format("csv") \
    .save("filepath")

I need to limit the size of the output file to 1gb.

For example, if the size of the data is 5gb, the output should be 5 files of 1 gb each.

How to achieve this?

pyspark apache-spark-sql apache-spark
1 Answers

You can repartition() the Dataframe before writing. It'll write one file per partition.

Following should produce 5 files.

df2 = df.repartition(5)
df2.write\
    .option("header",True) \
    .mode("overwrite") \
    .option("sep","|")\
    .format("csv") \
    .save("filepath")
user_4962890
Related