Partitioning strategy in Parquet and Spark

user_1511294 0
Viewed 761

I have a job that reads csv files , converts it into data frames and writes in Parquet. I am using append mode while writing the data in Parquet. With this approach, in each write a separate Parquet file is getting generated. My questions are :

  • 1) If every time I write the data to Parquet schema ,a new file gets appended , will it impact read performance (as the data is now distributed in varying length of partitioned Parquet files)
  • 2) Is there a way to generate the Parquet partitions purely based on the size of the data ?
  • 3) Do we need to think to a custom partitioning strategy to implement point 2? I am using Spark 2.3
parquet apache-spark
1 Answers

  1. It will affect read performance if spark.sql.parquet.mergeSchema=true.

    In this case, Spark needs to visit each file and grab schema from it.

    In other cases, I believe it does not affect read performance much.

  2. There is no way generate purely on data size. You may use repartition or coalesce. Latter will created uneven output files, but much performant.

    Also, you have config spark.sql.files.maxRecordsPerFile or option maxRecordsPerFile to prevent big size of files, but usually it is not an issue.

  3. Yes, I think Spark has not built in API to evenly distribute by data size. There are Column Statistics and Size Estimator may help with this.

user_64849300
Related