Spark deserialization of dataframe struct inside array of struct

user_3311616 0
Viewed 142

I have a dataframe schema (for data stored in parquet format) as below

root
 |-- id: string (nullable = true)
 |-- mid: integer (nullable = true)
 |-- relationship: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- child_id: string (nullable = true)
 |    |    |-- details: struct (nullable = true)
 |    |    |    |-- package_contains: string (nullable = true)
 |    |    |    |-- package_level: string (nullable = true)

If I have a query like

df.select($"id",$"mid").filter(array_contains(col("relationship.child_id"), "id1") && $"mid" === 1).show(false)

Does anyone know whether spark deserializes details struct inside the array of struct (relationship) when querying only for child_id column? and if there is a way to verify it?

parquet apache-spark-sql apache-spark
1 Answers

Spark won't look into the details struct unless it is necessary.

I create a new artifical dataset with the same structure. The details struct is calculated by an udf. Later, the udf is changed so that it throws an Exception at runtime. If the query still works, I know that the udf has not been called and the details struct has not been evaluated.

Attempt 1: the "nice" udf

def details(x:Int): Tuple2[String, String] = ("a", "b")
val detailsUdf = udf(details _)

val df = spark.range(start = 0, end = 5, step = 1)
  .withColumn("mid", lit(1))
  .withColumn("relationship",
      array(struct(concat(lit("id"),'id).as("child_id"), detailsUdf('id).as("details"))))

df now has the structure

root
 |-- id: long (nullable = false)
 |-- mid: integer (nullable = false)
 |-- relationship: array (nullable = false)
 |    |-- element: struct (containsNull = false)
 |    |    |-- child_id: string (nullable = false)
 |    |    |-- details: struct (nullable = true)
 |    |    |    |-- _1: string (nullable = true)
 |    |    |    |-- _2: string (nullable = true)

and contains the data

+---+---+---------------+
| id|mid|   relationship|
+---+---+---------------+
|  0|  1|[[id0, [a, b]]]|
|  1|  1|[[id1, [a, b]]]|
|  2|  1|[[id2, [a, b]]]|
|  3|  1|[[id3, [a, b]]]|
|  4|  1|[[id4, [a, b]]]|
+---+---+---------------+

Attempt 2: the bad udf

Now the udf is replaced by

def details(x:Int): Tuple2[String, String] = throw new NullPointerException
val detailsUdf = udf(details _)

The structure of the dataframe is still the same, but calling show now throws a NullPointerException.

If I run the query

df.select($"id",$"mid").filter(array_contains(col("relationship.child_id"), "id1") && $"mid" === 1).show(false)

I get the expected result

+---+---+
|id |mid|
+---+---+
|1  |1  |
+---+---+

After changing the query so that Spark has to evaluate the details struct

df.select($"id",$"mid").filter(array_contains(col("relationship.details._1"), "id1") && $"mid" === 1).show(false)

the query fails and throws a NullPointerException.

user_21298010
Related