Deserialize Avro Data from bytes

Viewed 4057

I am trying to deserialize, i.e., get an object of class org.apache.avro.generic.GenericRecord from byte array Avro data. This data contains a header with the full schema.

So far, I have tried this:

public List<GenericRecord> deserializeGenericWithSchema(byte[] message) throws IOException {
    List<GenericRecord> listOfRecords = new ArrayList<>();
    DatumReader<GenericRecord> reader = new GenericDatumReader<>();
    DataFileReader<GenericRecord> fileReader =
            new DataFileReader<>(new SeekableByteArrayInput(message), reader);

    GenericRecord record = null;

    while (fileReader.hasNext()) {
        listOfRecords.add(fileReader.next(record));
    }

    return listOfRecords;
}

But I am getting an error:

java.io.IOException: Invalid int encoding at

org.apache.avro.io.BinaryDecoder.readInt(BinaryDecoder.java:145) at

org.apache.avro.io.BinaryDecoder.readBytes(BinaryDecoder.java:282) at

org.apache.avro.file.DataFileStream.initialize(DataFileStream.java:112)

at org.apache.avro.file.DataFileReader.(DataFileReader.java:97)

However, if I write to disk the byte array message and change my function like:

public List<GenericRecord> deserializeGenericWithSchema(String fileName) throws IOException {
    byte[] file = new File(fileName);
    List<GenericRecord> listOfRecords = new ArrayList<>();
    DatumReader<GenericRecord> reader = new GenericDatumReader<>();
    DataFileReader<GenericRecord> fileReader =
            new DataFileReader<>(file, reader);

    GenericRecord record = null;

    while (fileReader.hasNext()) {
        listOfRecords.add(fileReader.next(record));
    }

    return listOfRecords;
}

It works flawlessly. I really don't want to write to disk every avro message I get because this is intended to work in a real time basis.

What am I doing wrong in my first approach?

0 Answers
Related