Pyspark Extract Values from from Array of maps in structured streaming

user_7989648 0
Viewed 3130

I have the following schema:

root
 |-- sents: array (nullable = false)
 |    |-- element: integer (containsNull = true)
 |-- metadata: array (nullable = true)
 |    |-- element: map (containsNull = true)
 |    |    |-- key: string
 |    |    |-- value: string (valueContainsNull = true)

In a table it looks like this:

+----------+---------------------------------------------------------------------+
|sents     |metadata                                                             |
+----------+---------------------------------------------------------------------+
|[1, -1, 0]|[[confidence -> 0.4991], [confidence -> 0.5378], [confidence -> 0.0]]|
+----------+---------------------------------------------------------------------+

How can I access the te Value from this list of maps within the array column?

thank you

spark-structured-streaming pyspark apache-spark nested python
1 Answers

Here are two options using explode and transform high-order function in Spark.

Option 1 (explode + pyspark accessors)

First we explode elements of the array into a new column, next we access the map using the key metadata to retrieve the value:

from pyspark.sql.functions import col, explode, expr

df = spark.createDataFrame([
  [[{"confidence":0.4991}, {"confidence":0.5378}, {"confidence":0.0}]]
], ["metadata"])

df.select(explode(col("metadata")).alias("metadata")) \
  .select(col("metadata")["confidence"].alias("value"))

# +------+
# |value |
# +------+
# |0.4991|
# |0.5378|
# |0.0   |
# +------+

Option 2 (transform + explode)

Here we use transform to extract the values of the map into a new array and then we explode it:

df.select(explode(expr("transform(metadata, i -> i['confidence'])")).alias("value"))
user_7503760
Related