Does spark.sql.autoBroadcastJoinThreshold work for joins using Dataset's join operator?

Viewed 67283

I'd like to know if spark.sql.autoBroadcastJoinThreshold property can be useful for broadcasting smaller table on all worker nodes (while making the join) even when the join scheme is using the Dataset API join instead of using Spark SQL.

If my bigger table is 250 Gigs and Smaller is 20 Gigs, do I need to set this config: spark.sql.autoBroadcastJoinThreshold = 21 Gigs (maybe) in order for sending the whole table / Dataset to all worker nodes?

Examples:

  • Dataset API join

    val result = rawBigger.as("b").join(
      broadcast(smaller).as("s"),
      rawBigger(FieldNames.CAMPAIGN_ID) === smaller(FieldNames.CAMPAIGN_ID), 
      "left_outer"
    )
    
  • SQL

    select * 
    from rawBigger_table b, smaller_table s
    where b.campign_id = s.campaign_id;
    
3 Answers

there is limitation that maximum size of DF that can be broadcast is 8G.

Related