How can I solve apache arrow and py4j running error?

Question

when I use apache arrow to transfor data from scala to python, I got a error I use py4j for communication with java and python

scala Code:

  def write(root: VectorSchemaRoot): Array[Byte] ={
    println(s"root count: ${root.getRowCount}")
    val out = new ByteArrayOutputStream()
    val writer = new ArrowStreamWriter(root, null, Channels.newChannel(out))
    writer.start()
    writer.writeBatch()
    writer.end()
    writer.close()
    val datas = out.toByteArray
    out.close()
    println(s"datas: ${datas.length}")
    datas
  }

Python Code:

 def transfrom(self):
        reader = pa.RecordBatchStreamReader(pa.BufferReader(self.data))
        pandas = reader.read_pandas()
        print(">>>>>>> data length: {}".format(len(pandas)))
        return pandas

I execute the code success in my macos, but in linux server I got a error:

Traceback (most recent call last):
  File "/data/yarn/nm/usercache/hive/appcache/application_1636515274577_0625/container_e191_1636515274577_0625_01_000003/python/lib/python3.6/site-packages/py4j/java_gateway.py", line 2451, in _call_proxy
    return_value = getattr(self.pool[obj_id], method)(*params)
  File "", line 33, in runner
  File "", line 25, in transfrom
  File "pyarrow/ipc.pxi", line 500, in pyarrow.lib._ReadPandasMixin.read_pandas
  File "pyarrow/ipc.pxi", line 563, in pyarrow.lib.RecordBatchReader.read_all
  File "pyarrow/error.pxi", line 114, in pyarrow.lib.check_status
OSError: Array length did not match record batch length

I find the data length have some problem:

In linux server:

data length: 652

I my macos:

datas: 66032

So, How can I solve this error?

How can I solve apache arrow and py4j running error?

0 Answers