Spark DataFrame: join stuck on the same stage

user_3951602 0
Viewed 613

I'm working on joining two large datasets with 17M and 2.2M row count. enter image description here

Partition size of the two datasets are:

mob_join_set:

enter image description here

dth_join_set:

enter image description here

Job is always stuck on the last two tasks of the join:

enter image description here

Have even tried G1GC and spark.sql.shuffle.partitions=500, spark.default.parallelism=500 But no success.

Any help will be greatly appreciated.

pyspark-sql pyspark apache-spark-sql apache-spark
0 Answers
Related