Are keys in RDDs same as indexes in dataframes?

user_17267424 0
Viewed 41

in pyspark, key value pairs are used to define a RDD. But are they conceptually same as indexes in dataframes?

rdd apache-spark key
1 Answers

RDDs in a more general way are a distributed collection of elements of data distributed across the cluster. Key/value pairs define a pair RDD, a type of RDD.

Spark Dataframes are inherently unordered so there is no concept of index like in pandas. Each row is treated as an independent collection of structured data.

user_116985620
Related