Repartitioned data bottlenecks in few tasks in Spark

user_17072058 0
Viewed 95

I have a simple spark job which does the following

val dfIn  = spark.read.parquet(PATH_IN)
val dfOut = dfIn.repartition(col1, col2, col3)
dfOut.write.mode(SaveMode.Append).partitionBy(col1, col2, col3).parquet(PATH_OUT)

I noticed in this job big performance deterioration. Inspecting the Spark UI showed that the write bottlenecked in a few tasks which showed huge memory spill and much bigger output size compared to the fast partitions.

So I suspected that this issue is caused by the data skew and changed the way the data is repartitioned to

import org.apache.spark.sql.functions.rand
val dfOut = dfIn.withColumn("rand", rand()).repartitionByRange(col1, col2, col3, $"rand")

however this did not help to resolve the performance issues. In the Spark UI you can see now that the data is very evenly distributed across ALL partitions (based on the output size). But still a few tasks are very long running.

I have no ideas what else could cause this and would be thankful for any ideas.

apache-spark-sql apache-spark
1 Answers

While this is not a final answer for your issue, this tip might help: you can easily inspect your actual data for possible skewness with

for i, part in enumerate(dfIn.rdd.glom().collect()):
    print({i: len(part)})

and then salting as needed. Of course all data might not fit, limit as appropriate to get a proper sample :)

PS: example in Python but you get the idea

user_59860360
Related