scala dataframe join columns and split arrays explode spark

user_16658261 0
Viewed 91

I have some co-ordinates in multiple array columns in a dataframe and want to split them to have the x,y,z in separate columns in order, column1 data first, then column 2

for example...

        COL 1                   |               COL2
[[x,y,z],[x,y,z],[x,y,z]...]    |   [[x,y,z],[x,y,z],[x,y,z]...]

e.g
    [[1,1,1],[2,2,2],[3,3,3]...]    |   [[8,8,8],[9,9,9],[10,10,10]...]

required OUTPUT

    COL X       |       COL Y       |       COL Z
x,x,x,x,x....   |   y,y,y,y,y....   |   z,z,z,z,z....

e.g.
 1,2,3,..,8,9,10..  |   1,2,3,..,8,9,10..   |   1,2,3,..,8,9,10..

any help appreciated

apache-spark-sql apache-spark dataframe scala
1 Answers

You can use array_union function as follows

df.select(
      array_union($"col1._1", $"col2._1").as("x"),
      array_union($"col1._2", $"col2._2").as("y"),
      array_union($"col1._3", $"col2._3").as("z"))

INPUT

+--------------------------------------------+--------------------------------------------------+
|col1                                        |col2                                              |
+--------------------------------------------+--------------------------------------------------+
|[[1, 1, 1], [2, 2, 2], [3, 3, 3], [4, 4, 4]]|[[8, 8, 8], [9, 9, 9], [10, 10, 10], [11, 11, 11]]|
+--------------------------------------------+--------------------------------------------------+

OTUPUT

+--------------------------+--------------------------+--------------------------+
|x                         |y                         |z                         |
+--------------------------+--------------------------+--------------------------+
|[1, 2, 3, 4, 8, 9, 10, 11]|[1, 2, 3, 4, 8, 9, 10, 11]|[1, 2, 3, 4, 8, 9, 10, 11]|
+--------------------------+--------------------------+--------------------------+
user_63783110
Related