how to manually run pyspark's partitioning function for debugging

user_6471140 0
Viewed 106

I'm having performance issues and after checking spark's web UI i found there is a severe data skewness issue: enter image description here

I have tried "distribute by" and "repartition" by multiple combinations of columns with no luck so i'm trying to debug how spark is partitioning the dataset(in order to fix it), is there any way to manually run the partitioning function for creating a column? Basically i'm trying to do something similar to:

df = df.withColumn("assigned_partition", partitioning_function())
df_grouped  = df.groupby("assigned_partition").count()

So that i can identify patterns or reasons for skewness.

Note: this is right after querying hive tables, so i know the skewness is not due to any spark logic or calculations.

pyspark apache-spark
1 Answers

I found it is possible to do it using: spark_partition_id() , then i created a histogram to analyze the count() distribution and found it looks normal and without outliers, so skewness is not from how the dataset is partitioned.

test_df = df.select(spark_partition_id().alias("partitionId"))

test_df.groupBy("partitionId").count().orderBy(col("count").desc()).select("count").toPandas().plot.hist()

plt.show()

enter image description here

user_64711400
Related