Change the datatype of any fields of Arraytype column in Pyspark

user_12613652 0
Viewed 81

I want to change the datatype of the field "value", which is inside the arraytype column "readings". The column "reading" has two fields, "key" nd "value".

root
 |-- name: string (nullable = true)
 |-- languagesAtSchool: array (nullable = true)
 |    |-- element: string (containsNull = true)
 |-- languagesAtSchool1: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- _id: integer (nullable = true)
 |-- languagesAtWork: array (nullable = true)
 |    |-- element: string (containsNull = true)
 |-- currentState: string (nullable = true)
 |-- previousState: double (nullable = true)
 |-- readings: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- value: integer (nullable = true)
 |    |    |-- key: string (nullable = true)

Expected Schema is

   root
     |-- name: string (nullable = true)
     |-- languagesAtSchool: array (nullable = true)
     |    |-- element: string (containsNull = true)
     |-- languagesAtSchool1: array (nullable = true)
     |    |-- element: struct (containsNull = true)
     |    |    |-- _id: integer (nullable = true)
     |-- languagesAtWork: array (nullable = true)
     |    |-- element: string (containsNull = true)
     |-- currentState: string (nullable = true)
     |-- previousState: double (nullable = true)
     |-- readings: array (nullable = true)
     |    |-- element: struct (containsNull = true)
     |    |    |-- value: string (nullable = true)
     |    |    |-- key: string (nullable = true)
pyspark apache-spark arrays
1 Answers

Transform using higher order function

Option 1; suitable when you want to drop some fields-name required fields instruct, sql expression

df1=df.withColumn('readings', expr('transform(readings, x-> struct(cast(x.value as integer) value,x.key))'))

or

Option 2; suitable when you dont want to name the fields in struct, also sql expression

df1=df.withColumn('readings', expr('transform(readings, x-> struct(x,cast(x.value as integer)))'))

option 3, suitable when you dont want to type fields in struct, doesnt use sql expression

df.withColumn('readings', F.transform('readings', lambda x: x.withField('value', x['value'].cast('int'))))


root
 |-- name: string (nullable = true)
 |-- languagesAtSchool: array (nullable = true)
 |    |-- element: string (containsNull = true)
 |-- languagesAtSchool1: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- id: integer (nullable = true)
 |-- languagesAtWork: array (nullable = true)
 |    |-- element: string (containsNull = true)
 |-- currentState: string (nullable = true)
 |-- previousState: double (nullable = true)
 |-- readings: array (nullable = true)
 |    |-- element: struct (containsNull = false)
 |    |    |-- value: integer (nullable = true)
 |    |    |-- key: string (nullable = true)
user_89869750
Related