Mapping and removing duplicates in Spark Scala?

user_4746684 0
Viewed 7035

I have a data set test.txt. It contain data like below

1::1::3
1::1::2
1::2::2
2::1::5
2::1::4
2::2::2
3::1::1
3::2::2

I have created data-frame using the below code. 

case class Rating(userId: Int, movieId: Int, rating: Float)
def parseRating(str: String): Rating = {
val fields = str.split("::")
assert(fields.size == 3)
Rating(fields(0).toInt, fields(1).toInt, fields(2).toFloat)
}

val ratings = spark.read.textFile("C:/Users/cravi/Desktop/test.txt").map(parseRating).toDF()

But when I am trying to print the output I am getting below output

[1,1,3.0]
[1,1,2.0]
[1,2,2.0]
[2,1,2.0]
[2,1,4.0]
[2,2,2.0]
[3,1,1.0]
[3,2,2.0]

But I want to print output like below I.e. removing duplicate combinations and instead of field(2) value 1.0.

[1,1,1.0]
[1,2,1.0]
[2,1,1.0]
[2,2,1.0]
[3,1,1.0]
[3,2,1.0]
apache-spark-sql apache-spark scala-collections scala
1 Answers

Once you have created a dataframe removing duplicates can be done by call .dropDuplicates(columnNames) and populating the third column with 1.0 can be done by using lit and withColumn functions.

To sum up, simple solution can be done as following

val ratings = sc.textFile("C:/Users/cravi/Desktop/test.txt")
    .map(line => line.split("::"))
    .filter(fields => fields.length == 3)
    .map(fields => Rating(fields(0).toInt, fields(1).toInt, 1.0f))
  .toDF().dropDuplicates("userId", "movieId")

ratings.show(false)

which should give you 

+------+-------+------+
|userId|movieId|rating|
+------+-------+------+
|3     |1      |1.0   |
|2     |2      |1.0   |
|1     |2      |1.0   |
|1     |1      |1.0   |
|2     |1      |1.0   |
|3     |2      |1.0   |
+------+-------+------+
user_58807060
Related