I am working on a Zeppelin Cluster (w Spark), using write.parquet(), I end up with multiple Parquet files.
I was wondering, is it possible to merge them into one file? Or I have to use a path="/folder/*" every time?
How to compress multiple HDFS files into one
Viewed 94
1 Answers
Use repartition():
df.repartition(1).write.parquet(path)
or, even better, coalesce()
df.coalesce(1).write.parquet(path)
Related