I am trying to use the broadcast join but getting below error in Spark 2.3. However, same code is working fine in Spark 2.2.
Upon checking the size of the dataframes its merely 50 MB and I have set the threshold to 200 MB as well. As I mentioned above same code is working fine in Spark 2.2
Error: "Cannot broadcast the table that is larger than 8GB".
However, Disabling the broadcasting is working fine.
'spark.sql.autoBroadcastJoinThreshold': '-1'
Could you please help here?