How to convert two columns to List[(Long,Int)] - Spark UDF

Viewed 22

I am new to Spark and trying to call an existing convertor method. I have a Spark DataFrame df with two columns. I want to convert these two columns to a List[(Long,Int)]

  val df = (spark.read.parquet(s"<file path>")
    .select(
      explode($"groups") as "groups"))
    .select(
      explode($"groups.lanes") as "lanes")
    .select(
      $"lanes.lane_path" as "lane_path")

Schema

root
|-- lane_path: struct (nullable = true)
|    |-- coordinates: array (nullable = true)
|    |    |-- element: long (containsNull = true)
|    |-- diffs: array (nullable = true)
|    |    |-- element: integer (containsNull = true)

UDF

val generateValues = spark.udf.register("my_udf",
       (laneAttrs: List[(Long,Int)], x: Long, y: Int) => {       
Converter.convert(laneAttrs, x, y)
      .map {
        case coord@Left(f) => throw new InterruptedException(s"$coord: $f")
        case Right(result) => Map("a" -> result.a,
          "b" -> result.b)
      }
  })

How can I pass "$lane_path.coordinates" and "$lane_path.diffs" as List[(Long,Int)] to call generateValues UDF?

0 Answers
Related