pyspark fillna is not working on column of ArrayType

user_2265190 0
Viewed 676

I have a spark cluster version 3.1.2. I have following input data

+-------+------+------------+
|   name|gender|         arr|
+-------+------+------------+
|  James|     M|     [60000]|
|Michael|     M| [70000, 31]|
| Robert|  null|[44, 400000]|
|  Maria|     F|[500000, 12]|
|    Jen|      |        null|
+-------+------+------------+

I have to remove null values from the all columns. "gender" column is StringType while "arr" column is of ArrayType. Some values are null is both columns. When I apply fillna function, values are remove from gender column but not from arr column. Have a look at output

>>> df.fillna("").show()
+-------+------+------------+
|   name|gender|         arr|
+-------+------+------------+
|  James|     M|     [60000]|
|Michael|     M| [70000, 31]|
| Robert|      |[44, 400000]|
|  Maria|     F|[500000, 12]|
|    Jen|      |        null|
+-------+------+------------+

Same happens if I use na.drop or na.fill functions. Where is the problem ? How can I remove null from arr column

pyspark apache-spark-sql apache-spark
1 Answers

fillna only supports int, float, string, bool datatypes, columns with other datatypes are ignored.

For example, if value is a string, and subset contains a non-string column, then the non-string column is simply ignored.(doc)

You can replace null values in array columns using when and otherwise constructs.

import pyspark.sql.functions as F
default_value = F.array().cast("array<int>")
fill_rule = F.when(F.col('arr').isNull(),default_value).otherwise(F.col('arr'))
df.withColumn('arr', fill_rule).show()
user_79895810
Related