I've been noticing something, and wanted to get an explanation for it (or at least confirmation.)
My company has a common pattern where in order to do something to all of the columns of a DataFrame, we do
colList.foldleft(df)({case(df_, colName) => df_.withColumn...})
This works in general, but I've found some cases where it's very slow, if working on a DataFrame with a bunch of columns and a complex upstream. What's interesting is that it's slow even before an action is applied; a not-super-long sequence of transformations can take 4-14 minutes to evaluate.
If I re-write the same code to use
df.select(df.columns.map(c => ....)), the same code takes ~2 seconds. Calling .explain() on either produces the same output.
Can anyone explain this?