I am trying to establish an IPC pipeline. I have a python program which is writing an .arrow file and saving it in memory which gets picked up by the Java application. This application reads the file schema and does relevant operations. Currently I am having trouble with a column that is of float16 datatype. So my python program writes this column something like this:
# sample
float16column= pa.array(([np.float16(0) for _ in range(5)]), type=type_float16)
item_table = pa.table([float16column], ['samplefloat16columnname'])
local = fs.LocalFileSystem()
with local.open_output_stream("output.arrow") as file:
with pa.RecordBatchFileWriter(file, table.schema) as writer:
writer.write_table(table)
Now when I try to read this file from java application (and mind you this is the only column throwing error) using the below program
public void read(String path) throws IOException {
File arrowFile = new File(path);
FileInputStream fileInputStream = new FileInputStream(arrowFile);
SeekableReadChannel seekableReadChannel = new SeekableReadChannel(fileInputStream.getChannel());
ArrowFileReader arrowFileReader = new ArrowFileReader(seekableReadChannel,
new RootAllocator(Integer.MAX_VALUE));
List<ArrowBlock> arrowBlocks = arrowFileReader.getRecordBlocks();
for (int i = 0; i < arrowBlocks.size(); i++) {
ArrowBlock rbBlock = arrowBlocks.get(i);
if (!arrowFileReader.loadRecordBatch(rbBlock)) { // load the batch
throw new IOException("Expected to read record batch");
}
// do something with the loaded batch
}
}
I see this error:
Exception in thread "main" java.lang.UnsupportedOperationException: NYI: FloatingPoint(HALF)
Now I am not very proficient in java, but I am guessing this may have something to do with incompatible data types of both. Does anyone else know the correct of way of doing this ?
ps: reading the same arrow file using python seems to be working fine