i have an issue ,
in my compagny for reading an avro file we do Something like this:
``` val rdd = spark.sparkContext.binaryFiles(path)
implicit val streamEncoder: Encoder[(String, PortableDataStream)] = Encoders.kryo[(String, PortableDataStream)]
spark.createDataset(rdd) ```
this format allow us to easily push the data to a server, without changing the avro, but now we need to modify the avro file for adding to it some new columns before send it to server, by doing something like this :
val avroDF2 = spark.read.avro(restitutionPath2)
val df = avroDF2.select(col("*") +:predefinedList.map(x => lit("").alias(x)): _*)
now my problem is that i don't find some way to convert my dataframe (df) to an RDD[(string,PortableDatastream)] in order to directy push it to my server instead of writing in the avro