Why is Spark broadcasting 1.5GB table (see DAG)?

Viewed 104

I am using Spark 2.3.1.

config while initializing spark: config("spark.sql.autoBroadcastJoinThreshold",'52428800') #50M

Spark is broadcasting a table of around 1.5G in size. What could be the cause?

Is the data under 50M after compression? The broadcast is causing delays/even killing of job.

DAG from SQL tab Spark Web UI, look at bottom right

enter image description here The pyspark code I am using (no explicit broadcast being used):

(
    a
    .join(b,'col1','left')
    .join(c,'col2','left')
)
0 Answers
Related