Need to check if event messages sent to Kafka are valid by checking if message has needed fields and if so, push data to Elasticsearch. This is how I done it:
object App {
val parseJsonStream = (inStream: RDD[String]) => {
inStream.flatMap(json => {
try {
val parsed = parse(json)
Option(parsed)
} catch {
case e: Exception => System.err.println("Exception while parsing JSON: " + json)
e.printStackTrace()
None
}
}).flatMap(v => {
if (v.values.isInstanceOf[List[Map[String, Map[String, Any]]]])
v.values.asInstanceOf[List[Map[String, Map[String, Any]]]]
else if (v.values.isInstanceOf[Map[String, Map[String, Any]]])
List(v.values.asInstanceOf[Map[String, Map[String, Any]]])
else {
System.err.println("EVENT WRONG FORMAT: " + v.values)
List()
}
}).flatMap(mapa => {
val h = mapa.get("header")
val b = mapa.get("body")
if (h.toSeq.toString.contains("session.end") && !b.toSeq.toString.contains("duration")) {
System.err.println("session.end HAS NO DURATION FIELD!")
None
}
else if (h.isEmpty || h.get.get("userID").isEmpty || h.get.get("timestamp").isEmpty) {
throw new Exception("FIELD IS MISSING")
None
}
else {
Some(mapa)
}
})
}
val kafkaStream: InputDStream[ConsumerRecord[String, String]] = KafkaUtils.createDirectStream[String, String](
ssc, PreferBrokers, Subscribe[String, String](KAFKA_EVENT_TOPICS, kafkaParams)
)
val kafkaStreamParsed = kafkaStream.transform(rdd => {
val eventJSON = rdd.map(_.value)
parseJsonStream(eventJSON)
}
)
val esEventsStream = kafkaStreamParsed.map(addElasticMetadata(_))
try {
EsSparkStreaming.saveToEs(esEventsStream, ELASTICSEARCH_EVENTS_INDEX + "_{postfix}" + "/" + ELASTICSEARCH_TYPE, Map("es.mapping.id" -> "docid")
)
} catch {
case e: Exception =>
EsSparkStreaming.saveToEs(esEventsStream, ELASTICSEARCH_FAILED_EVENTS)
e.printStackTrace()
}
}
I guess someone is sending invalid events (that's the reason why am I doing this check anyway), but Spark job doesn't skip the message, it fails with message:
User class threw exception: org.apache.spark.SparkException: Job aborted due to stage failure: Task 2 in stage 6.0 failed 4 times, most recent failure: Lost task 2.3 in stage 6.0 (TID 190, xxx.xxx.host.xx, executor 3): java.lang.Exception: FIELD IS MISSING
How can I prevent it from crashing and to just skip message instead? It is YARN application, using:
Spark 2.3.1
Spark-streaming-kafka-0-10_2.11:2.3.1
Scala 2.11.8