It looks RDD is to be removed from Spark.
Announcement: DataFrame-based API is primary API
The RDD-based API is expected to be removed in Spark 3.0
Then, how to implement programs like word count in Spark?
Spark - How to word count without RDD
Viewed 32
1 Answers
The data you manipulate as tuples using RDD api can be thought of and manipulated as columns/fields in a SQL like manner using DataFrame api.
df.withColumn("word", explode(split(col("lines"), " ")))
.groupBy("word")
.count()
.orderBy(col("count").desc())
.show()
+---------+-----+
| word|count|
+---------+-----+
| foo| 5|
| bar| 2|
| toto| 1|
...
+---------+-----+
Notes:
- This code snippet requires necessary imports from
org.apache.spark.sql.functions - Relevant examples can be found in this question's answers.
Related