Output csv instead of snappy.parquet

user_9135031 0
Viewed 205

I am running the below:

  for x in days:
       for i in numbers:
           mega_query = spark.sql("select *  " \
                        "from db.table " \
                        "where date ={} " \
                        "and number = {} ".format(x, 1))

     
 
  mega_query.write.mode('append').options(header='true').save("/path/results2.csv")

I want to have one csv file at the end but instead of that I have many snappy.parquet files. How can I create only one csv with the results?

pyspark apache-spark python-3.x python
2 Answers

you need to add .format("csv") into your .write command. Otherwise it will use default Parquet format.

If you want to have one file, then you can add .coalesce(1) before .write, but this may not be the most optimal solution from performance standpoint

Alex_Ott_186270

daria]1 API, to create a single csv file with a specific filename.

import com.github.mrpowers.spark.daria.sql.DariaWriters

DariaWriters.writeSingleFile(
    df = df,
    format = "csv",
    sc = spark.sparkContext,
    tmpFolder = sys.env("HOME") + "/Documents/better/tmp",
    filename = sys.env("HOME") + "/Documents/better/mydata.csv"
)

You can find more details in the following tutorial.

https://mungingdata.com/apache-spark/output-one-file-csv-parquet/

user_19189620
Related