Add new columns

user_13559372 0
Viewed 1962

ErrorHi I am trying to a new column to a Spark. I am trying in a data set where I want to add the percentage made by in all games. The data set looks like this:

Name, Platform, Year, Genre, Publisher, NA_Sales, EU_Sales, JP_Sales, Other_Sales

val vgdataLines = sc.textFile("hdfs:///user/ashhall1616/bdc_data/t1/vgsales-small.csv")
val vgdata = vgdataLines.map(_.split(";"))
def toPercentage(x: Double): Double = {x * 100} val countPubl  = vgdata.map(r =>  (r(4),1)).reduceByKey(_+_)
val addpercen = countPubl.withColumn("count", toPercentage($"count"/countPubl.count(_._2)))

I used withColumn() to add new column 'count' and expected output to be like:

(Ubisoft,3,15.0)

Can anyone tell whats wrong here?

scala
2 Answers

You cannot use "withColumn" with an RDD. You could do as follow

val addpercen = countPubl.map({case(key, value) => (key, value, toPercentage(value))})

use map to add a calculated value as new column and convert to a DataFrame if you want

import spark.implicits._
val myDf = addpercen.toDF("key","value","myNewColumn") 

myDf.show()

Hope it helps.

user_85714980

You can not use withColumn with an RDD hence convert it to DataFrame as below and then use it

val countPubl : DataFrame  = vgdata.map(r =>  (r(4),1)).reduceByKey(_+_).toDF()

If you still looking to use RDD then just converto it back to RDD once you add the with column as

val javaRdd : JavaRDD[Row] = countPubl.withColumn("...",col("...")).toJavaRDD
user_63035790
Related