Databricks - how is number of partitions determined?

user_14238391 0
Viewed 346

I'm processing a file for which I run a repartition(48, key) prior to saving as a delta. At initial run, it generates around 25 partitions within the delta (no issue as it's possible the key resulted in data falling into 25 partitions - I assume it doesn't necessarily create a partition for a node that didn't have data?).

However, on the second run, which is done via a MERGE, 60+ partition files were generated (it's a SCD process with 1700 keys pre-existing and only 300 new keys)

My understanding is that repartition uses a hash algorithm to determine which of the partitions a key falls into, so if that is the case, how did it create more partition files than defined partitions?

Code sample:

df = spark.read...
  
key = ["COL_A"]
partitions = 48
df.repartition(partitions, *[col(c) for c in key])
delta-lake databricks apache-spark
1 Answers

The "repartition" operation was on the original data frame, after it is converted to delta-lake it doesn't hold on to those repartition settings. Once delta-lake is created, the merge operation will find the partitions which match the whenMatched condition and just replace them with new data. As you saw it skipped a lot of data and improved performance. Similarly you could add/append new data and that may land as new partitions as well. You need to Optimize yor delta-lake to consolidate small files created as new data is changed.

user_37466370
Related