spark Dataframe vs pandas-on-spark Dataframe

user_14566962 0
Viewed 1593

I'm quite new to pySpark but I'm confused about the difference between a spark Dataframe (created for example from an RDD ) and a pandas-on-spark Dataframe.

  • Are those the same object ? Looking at the type it seems they are different classes.
  • What's the core difference, if any ? (I know that working with pandas-on-spark Dataframe you can use almost the same syntax of Pandas on a distributed Dataframe but I'm wondering if is only this one the difference )

Thanks

pyspark apache-spark-sql
1 Answers

Answering directly:

Are those the same object ? Looking at the type it seems they are different classes.

No, they are completely different objects (classes).

What's the core difference, if any ?

A pySpark DataFrame is an object from the PySpark library, with its own API and it can be constructed from a wide array of sources such as: structured data files, tables in Hive, external databases, or existing RDDs.

A Pandas-on-Spark DataFrame and pandas DataFrame are similar. However, the former is distributed and the latter is in a single machine. When converting to each other, the data is transferred between multiple machines and the single client machine.

A Pandas DataFrame, is an object from the pandas library, also with its own API and it can also be constructed from a wide range of methods.

Also, I recommend checking this documentation about Pandas on Spark

user_135265120
Related