How is the number of partitions for an inner join calculated in Spark?

user_2428401 0
Viewed 596

We have two dataframes. df_A and df_B

  • df_A.rdd.getPartitionsNumber() # => 9

  • df_B.rdd.getPartitionsNumber() # => 160

  • df_A.createOrReplaceTempView('table_A')

  • df_B.createOrReplaceTempView('table_B')

After creation joined dataframe via SparkSQL,

  • df_C = spark.sql(""" select * from table_A inner table_B on (...) """)
  • df_C.rdd.getPartitionsNumber() # => 160

How does Spark calculate and use these two partitions for two joined dataframes?

Shouldn't the number of partitions of the joined dataframe be 9 * 160 = 1440?

bigdata aws-glue apache-spark
1 Answers

Spark configures the number of partitions to 200 when shuffling data for joins or aggregations. You can change the value in spark.sql.shuffle.partitions to increase or decrease the number of partitions in join operation.

https://spark.apache.org/docs/latest/sql-performance-tuning.html
user_86086030
Related