When a column is never used in an action in spark sql, is it ever loaded into memory?

In most cases spark's optimizer should be smart enough to only load needed columns (projection pushdown). This is especially powerful for columnar file formats such as parquet, it means that not the entire file(s) needs to be read.

But I think in some cases its still better to make an explicit select statement, I think I have seen cases where the projection pushdown did not work well enough, mostly in very complex queries.

To be sure, use spark's df.explain