How do I convert Binary string to scala string in spark scala

user_8104738 0
Viewed 15955

I am reading an avro file which contains a field as binary string, I need to convert it into a java.lang.string to pass it to another library(spark-xml-util), how do I convert it into java.lang.string efficiently. This is the code I have got so far : -

    val df = sqlContext.read.format("com.databricks.spark.avro").load("filePath/fileName.avro")
    df.select("myField").collect().mkString

The last line gives me the following exception: -

Exception in thread "main" java.lang.ClassCastException: [B cannot be cast to java.lang.String
    at org.apache.spark.sql.Row$class.getString(Row.scala:255)
    at org.apache.spark.sql.catalyst.expressions.GenericRow.getString(rows.scala:165)

df schema is: -

root
|-- id: string (nullable = true)
|-- myField: binary (nullable = true)
spark-dataframe apache-spark scala string
3 Answers

In Spark 3.0 you can cast between BINARY and STRING data.

scala> val df = sc.parallelize(Seq("ABC", "BCD", "CDE", "DEF")).toDF("value")
df: org.apache.spark.sql.DataFrame = [value: string]

scala> df.select($"value", $"value".cast("BINARY"), 
$"value".cast("BINARY").cast("STRING")).show()
+-----+----------+-----+
|value|     value|value|
+-----+----------+-----+
|  ABC|[41 42 43]|  ABC|
|  BCD|[42 43 44]|  BCD|
|  CDE|[43 44 45]|  CDE|
|  DEF|[44 45 46]|  DEF|
+-----+----------+-----+

I don't have your data to test with, but you should be able to do:

df.select($"myField".cast("STRING"))

This obviously depends on the actual data (i.e. don't cast a JPEG to STRING) but assuming it's UTF-8 encoded this should work.

user_111688220

In the previous solution, the code new String(payload) did not work for me on true binary data.

Ultimately the solution was a little more involved, with the length of the binary data required as a 2nd parameter.

def binToString(payload: Array[Byte], payload_length: Int): String = {
  val ac: Array[Char] = Range(0,payload_length).map(i => payload(i).toChar).toArray
  return ac.mkString 
}

val binToStringUDF = udf( binToString(_: Array[Byte], _: Int): String )
user_30776730
Related