Spark-Scala RDD, group by count from array of array

user_18809216 0
Viewed 107

I'm fairly new to Spark-Scala and just started to learn it by myself so please bear with me, I am using oracle virtual machine.

Here is my code,

val dataLines = sc.textFile("Data/client_jobs.csv")
val data = dataLines.map(_.split(";"))

val values = data.map(array => (array(0), array(1))

I can fetch data using spark-shell, and get the data in array of array format like this,

val data = Array[Array[String]] = Array(Array("c1",20)
                                ,Array("c2",102)
                                ,Array("c3",50)
                                ,Array("c4",80)
                                ,Array("c5",140)
                                ,Array("c6",2036), Array("c7",568))

As you can see from the code I have also mapped it but instead of giving me an output like this

Array(("c1",20), ("c2",102), ("c3",50)...)

It gives me,

MapPartitionsRDD[3] at map at code1.scala:14

From the array of array or the normal array I need to fetch data and get the output like shown below,

(below100, 3)

(100 to 150, 2)

(above150, 2)

Basically it is only counting the jobs within the range.

I know very little about Spark-Scala, so any kind of help is appreciated. Thanks in advance.

rdd apache-spark scala
1 Answers

If you have RDD of tuples, category can be applied to each value, and then grouped:

val data = Array(("c1", 20),
  ("c2", 102),
  ("c3", 50),
  ("c4", 80),
  ("c5", 140),
  ("c6", 2036),
  ("c7", 568))

val rdd = sparkContext.parallelize(data)

rdd.map(tuple =>
  (
    if (tuple._2 < 100) {
      "below100"
    }
    else if (tuple._2 >= 100 && tuple._2 <= 150) {
      "100 to 150"
    }
    else "above150"
    , 1
  )
).reduceByKey(_ + _)

Result is:

(below100,3)
(above150,2)
(100 to 150,2)
user_26673430
Related