I have faced some out of memory issues in spark and most solutions tell me to reduce the collect() operations or check broadcast tables.
So I have a simple question, why does this happen when I am not using collect or broadcast tables in my code?
Dose spark perform collect in the intermediate processing for some operations?