does df.write.csv create number of partitions equals to total no of records in the df?

rashmi jadhao 0
Viewed 17

add_dist.write.format("csv").option("sep",",").mode("overwrite").save("C:\BigData\projects\datalake\address_op") i am trying to write into the folder in csv format using pyspark.

Dataframe has 25 total records and it is creating 25 part00000-part00024 partitions in the folder after writing.... what do i do to get all in single file(partition)

partition format dataframe pyspark apache-spark-sql
1 Answers

It's more efficient to use coalesce instead of repartition in this case.

Here is a function that might help. With this function, you can also define the file name:

def export_csv(df, fileName, filePath):
  
  filePathDestTemp = filePath + ".dir/" 

  df\
    .coalesce(1)\
    .write\
    .save(filePathDestTemp) 

  listFiles = dbutils.fs.ls(filePathDestTemp)
  for subFiles in listFiles:
    if subFiles.name[-4:] == ".csv":
      
      dbutils.fs.cp (filePathDestTemp + subFiles.name,  filePath + fileName+ '.csv')

  dbutils.fs.rm(filePathDestTemp, recurse=True)
Luiz Viola0
Related