I am using Spark 2.3.1.
config while initializing spark: config("spark.sql.autoBroadcastJoinThreshold",'52428800') #50M
Spark is broadcasting a table of around 1.5G in size. What could be the cause?
Is the data under 50M after compression? The broadcast is causing delays/even killing of job.
DAG from SQL tab Spark Web UI, look at bottom right
The pyspark code I am using (no explicit broadcast being used):
(
a
.join(b,'col1','left')
.join(c,'col2','left')
)