Relatively new to spark and Curious with the behaviour of PySpark wherein ,
While performing an inner join (According to execution plan :SortMergeJoin ) using PySpark , with following parameters enabled :
spark.sql.adaptive.enabled true
spark.sql.adaptive.coalescePartitions.enabled true
spark.sql.adaptive.skewJoin.enabled true
I see that there are partitions which are greater than spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes (256 MB the default value) during job execution.
Referring to this
The Median of partition sizes come upto 18MB . spark.sql.adaptive.skewJoin.skewedPartitionFactor is also set to default value 5
However there are 2 partitions existing with sizeable skew :1454.9 MiB and 791.5 MiB respectively .
Isn't the above parameters supposed to ensure that these 2 partitions get split to smaller ones to avoid data skew ?