Spark save and read Array[Byte] type

user_15040965 0
Viewed 190

I have serialize the object to Array[Byte] type, and save it to the parquet file as StructField("byteArrayObject",ArrayType(ByteType), nullable = true). When I try to read it, using row.getAs[Array[Byte]]("byteArrayObject") there is an error:

scala.collection.mutable.WrappedArray$ofRef cannot be cast to [B

Any one know what the problem is?

apache-spark scala
1 Answers

Spark deserializes array values as WrappedArray. Try the following:

import scala.collection.mutable.WrappedArray
import java.{lang => jl}

row
  .getAs[WrappedArray[jl.Byte]]("byteArrayObject")
  .map(_.byteValue)
  .array
user_13744610
Related