Why is spark dataframe repartition faster than coalesce when reducing number of partitions?

user_1611228 0
Viewed 983

I have a df with 100 partitions, and before saving to HDFS as .parquet I want to reduce the number of partitions because the parquet files would be too small (<1MB). I've added coalesce before writing:

df.coalesce(3).write.mode("append").parquet(OUTPUT_LOC)

It works but slows down the process from 2-3s per file to 10-20s per file. When I try repartition:

df.repartition(3).write.mode("append").parquet(OUTPUT_LOC)

The process does not slow down at all, 2-3s per file.

Why? Shouldn't coalesce always be faster when reducing the number of partitions because it avoids a full shuffle?

Background:

I'm importing files from local storage to spark cluster and saving the resulting dataframes as a parquet file. Each file is approx 100-200MB. Files are located on the "spark-driver" machine, I'm running spark-submit in client deploy mode. I'm reading files one by one in driver:

data = read_lines(file_name)
rdd = sc.parallelize(data,100)
rdd2 = rdd.flatMap(lambda j: myfunc(j))
df = rdd2.toDF(mySchema)
df.repartition(3).write.mode("append").parquet(OUTPUT_LOC)

Spark version is 3.1.1

Spark/HDFS cluster has 5 workers with 8CPU,32GB RAM

Each executor has 4cores and 15GB RAM, that makes 10 executors total.

EDIT:

When I use coalesce(1) I get spark.rpc.message.maxSize limit breached error, but not when I use repartition(1). Could that be a clue?

Attaching DAG visualizations .. Looks like WholeStageCodegen part is taking too long on coalesce DAGs?

Coalesce

Coalesce Query

Repartition

Repartition Query

hdfs apache-spark
1 Answers

This can happen sometimes if your data is not evenly distributed and when you do coalesce it tries to reduce the partitions by combining the small partitions in order to reduce full shuffle but there could still be some data skew in one of the partition and that single partition would be taking the most of the time.

While you do repartition the data gets distributed almost evenly on all the partitions as it does full shuffle and all the tasks would almost get completed in the same time.

You could use the spark UI to see why when you are doing coalesce what is happening in terms of tasks and do you see any single task running long.

user_48788480
Related