Would like to know how to convert Dataset<Row> to List<GenericRecord>.
I'm speaking on:
org.apache.avro.generic.GenericRecord
org.apache.spark.sql.Dataset
org.apache.spark.sql.Row
Dataset<Row> data = spark.sql(SQL_QUERY)
The result is different per SQL_QUERY, therefore the schema can be different per use case.
Important to know that I'm reading from an Iceberg table, saving files as .avro under the hood.
My current thinking is to find a way to convert each Row of the Dataset<Row> to bytes[] and then to:
public static List<GenericRecord> deserialize(byte[] bytes) {
List<GenericRecord> records = new ArrayList<>();
try {
DataFileReader<GenericRecord> reader = new DataFileReader<>(
new SeekableByteArrayInput(bytes),
new ExpectedSpecificDatumReader()
);
while (reader.hasNext()) {
records.add(reader.next(null));
}
reader.close();
} catch (Exception e) {
throw new Error(e);
}
return records;
}
Would appreciate your help here :)