How to repartition spark dataframe into smaller partitions

user_7173119 0
Viewed 70

I have a dataframe which is partitioned by date. In normal processing, I am processing a week of data at a time, so this means I have 7 partitions. I would like to increase this number of partitions, but without having to shuffle data or have a mix of dates in the same partition.

I've tried using df.repartition(20, my_date_column), but this just results in 13 empty partitions since the hash partitioner will only get 7 distinct values.

I've also tried using df.repartition(20, my_date_column, unique_id), which does increase the number of partitions to 20, but it means that dates are mixed within the partitions.

Is what I'm trying to do possible?

pyspark apache-spark-sql apache-spark
1 Answers

Perhaps you can increase the number of partitions by setting spark.sql.files.maxPartitionBytes to a smaller value. According to the tuning guide:

Property Name: spark.sql.files.maxPartitionBytes
Default: 134217728 (128 MB)
Meaning: The maximum number of bytes to pack into a single partition when reading files. This configuration is effective only when using file-based sources such as Parquet, JSON and ORC.
Since Version: 2.0.0

Alternatively, you can try spark.sql.files.minPartitionNum but it only means to suggest, not guarantee, the number of partitions.

user_6387640
Related