Transfer Scala case class to JsValue in rdd.map func but Task not serializable

user_13723952 0
Viewed 132

I am new to Scala/Spark and I have RDD of case class 

case class Info(key1 : String, key2 : String, key3 : String)

I want to transfer RDD[Info] into RDD[JsString] and save it to ElasticSearch, I use play.api.libs and define write converter:

implicit val InfoWrites = new Writes[Info]{
   def writes(i : Info): JsObject = Json.obj( 
     "key1" -> i.key1,
     "key2" -> i.key2,
     "key3" -> i.key3
   )
}

then I define implicit class to use save func:

implicit class Saver(rdd : RDD[Info]) {
    def save() : Unit = {
       rdd.map{ i => Json.toJson(i).toString }.saveJsonToEs("resource"))
    }
}

So I can save RDD[Info] with 

infoRDD.save()

But I keep get the "Task not serializable" error with Json.toJson() in rdd.map()

I also try to define serializeable object like this 

object jsonUtils extends Serializable{
   def toJsString(i : Info) : String = {
       Json.toJson(i).toString()
   }
}
rdd.map{ i => jsonUtils.toJsString(i) }

but keep getting error "Task not serializable"
How to change the code ? Thank you !

rdd elasticsearch apache-spark scala
1 Answers

I ran the below code, similar to your code and it works for me:

import models.Info
import org.apache.spark.rdd.RDD
import play.api.libs.json.Json
import domain.utils.Implicits._

class CustomFunctions(rdd : RDD[Info]) {
  def save() = {
    rdd.map(i => Json.toJson(i).toString ).saveAsTextFile("/home/training/so-123")
  }

}

Wrote the corresponding Implicits:

package domain.utils

import play.api.libs.json.{JsObject, Json, Writes}
import models.Info

class Implicits {
  implicit val InfoWrites = new Writes[Info]{
    def writes(i : Info): JsObject = Json.obj(
      "key1" -> i.key1,
      "key2" -> i.key2,
      "key3" -> i.key3
    )
  }

}

object Implicits extends  Implicits

Created the model Info:

package models

case class Info(key1 : String, key2 : String, key3 : String)

Created a SparkOperationsDao to compose and create spark context:

package dao

import domain.utils.CustomFunctions
import models.Info
import org.apache.spark.{SparkConf, SparkContext}


class SparkOperationsDao {
  val conf:SparkConf = new SparkConf().setAppName("driverTrack").setMaster("local")
  val sc = new SparkContext(conf)

  def writeToElastic() = {
    val sample = List(Info("name1", "city1", "123"), Info("name2", "city2", "234"))
    val rdd = sc.parallelize(sample)
    val converter = new CustomFunctions(rdd)
    converter.save()
  }

}

object SparkOperationsDao extends SparkOperationsDao

Run the App:

import dao.SparkOperationsDao

object RapidTests extends App {
  SparkOperationsDao.writeToElastic()
    //.collect.foreach(println)

}
user_41608890
Related