Consider we have a spark job having 5 executors, each having a memory of 5 GB and driver with 1 GB. We have a df having volume of 20 MB, now for the purpose of joining we have broadcasted it, now as the data is distributed over executors, data will be collected at the driver end and then it will be sent to all executors for joining purpose.
So what will be the data volume of the df after it has been broadcasted, will it be 120 MB ? and after the join operation is completed how long before the data is cleared from the memory? will it be immediate or it will wait for next GC cycle ?