split a column in Scala

Question

Hi I am trying to split a column in spark.

Data set sample:

Here I want to split the Month column to a Month and a year: Example:

2019 10

2009 11

and further count all the tweets in a year.(I know how to use reduceByKey(+) here)

How do I split columns in Spark? I don't want to use Data frames.

user_8571498 · Answer

You can try as follow

val rdd = oldRdd.map({case(tokenType,month,count,hashTagName) => (tokenType,month.substring(0,4),month.substring(2,6),count,hashTagName)})