How to handle null values when concatenating Maptype values in Spark

user_16413909 0
Viewed 306

I am trying to concatenate two columns of Maptype using concat_map(). My problem is when I try to concat a null and a Map I get a null when I expect to get the nonNull Map value.

val DF_concatenated=    DF.select(col("_1"), map_concat(col("m2"),col("m3"))).show()

I am trying to get from this DataFrame DF:

+---+----------+----------------+
| _1|        m2|              m3|
+---+----------+----------------+
|  3|[c -> III]|            null|
|  1|  [a -> I]|     [one -> un]|
|  4|      null|[four -> quatre]|
|  2| [b -> II]|   [two -> deux]|
+---+----------+----------------+

to this Dataframe DF_concatenated:

+---+----------------------+
| _1|  map_concat(m2, m3)  |
+---+----------------------+
|  3|           [c -> III] |
|  1| [a -> I, one -> un]  |
|  4|    [four -> quatre]  |
|  2|[b -> II, two -> deux]|
+---+----------------------+

but I end up having this output:

+---+----------------------+
| _1|  map_concat(m2, m3)  |
+---+----------------------+
|  3|                null  |
|  1| [a -> I, one -> un]  |
|  4|                null  |
|  2|[b -> II, two -> deux]|
+---+----------------------+
apache-spark null scala
1 Answers

The behaviour of map_concat is that if even a single operand is null it returns null.

If your columns are nullable you could use coalesce to substitute null with an empty map.

DF.select(
   col("_1"),
   map_concat(
       coalesce(col("m2"), map()),
       coalesce(col("m3"), map())
   ).as("result")
).show()
user_13033240
Related