I am in Spark, I have an RDD from an Avro file. I now want to do some transformations on that RDD and save it back as an Avro file:
val job = new Job(new Configuration())
AvroJob.setOutputKeySchema(job, getOutputSchema(inputSchema))
rdd.map(elem => (new SparkAvroKey(doTransformation(elem._1)), elem._2))
.saveAsNewAPIHadoopFile(outputPath,
classOf[AvroKey[GenericRecord]],
classOf[org.apache.hadoop.io.NullWritable],
classOf[AvroKeyOutputFormat[GenericRecord]],
job.getConfiguration)
When running this Spark complains that Schema$recordSchema is not serializable.
If I uncomment the .map call (and just have rdd.saveAsNewAPIHadoopFile), the call succeeds.
What am I doing wrong here?
Any idea?