Spark dataframe: collect () vs select ()

To answer the questions directly:

Will collect() behave the same way if called on a dataframe?

Yes, spark.DataFrame.collect is functionally the same as spark.RDD.collect. They serve the same purpose on these different objects.

What about the select() method?

There is no such thing as spark.RDD.select, so it cannot be the same as spark.DataFrame.select.

Does it also work the same way as collect() if called on a dataframe?

The only thing that is similar between select and collect is that they are both functions on a DataFrame. They have absolutely zero overlap in functionality.

Here's my own description: collect is the opposite of sc.parallelize. select is the same as the SELECT in any SQL statement.

If you are still having trouble understanding what collect actually does (for either RDD or DataFrame), then you need to look up some articles about what spark is doing behind the scenes. e.g.:

• https://dzone.com/articles/how-spark-internally-executes-a-program
• https://data-flair.training/blogs/spark-rdd-operations-transformations-actions/

Select is a transformation, not an action, so it is lazily evaluated (won't actually do the calculations just map the operations). Collect is an action.

Try:

df.limit(20).collect()

Short answer in bolds:

• collect is mainly to serialize
  (loss of parallelism preserving all other data characteristics of the dataframe)
  For example with a PrintWriter pw you can't do direct df.foreach( r => pw.write(r) ), must to use collect before foreach, df.collect.foreach(etc).
  PS: the "loss of parallelism" is not a "total loss" because after serialization it can be distributed again to executors.

• select is mainly to select columns, similar to projection in relational algebra
  (only similar in framework's context because Spark select not deduplicate data).
  So, it is also a complement of filter in the framework's context.

Commenting explanations of other answers: I like the Jeff's classification of Spark operations in transformations (as select) and actions (as collect). It is also good remember that transforms (including select) are lazily evaluated.