how to convert columns to rows in pyspark?

user_16053370 0
Viewed 110

I have a dataframe with several columns, and I need to put it in a single column, how can I do it performatically without losing the parallelism power of pyspark? Can someone help me?

USER|ID_1   |ID_2   |ID_3   |ID_4   |Name1|Name2|Name3|Name4

1234|1234500|null   |null   |null   |text |null |null |null   
1234|1234500|null   |null   |null   |text |null |null |null   
1234|null   |null   |1111500|null   |null |null |text |null   
1234|null   |null   |1111500|null   |null |null |text |null   
1234|null   |1111222|null   |null   |null |text |null |null   
1234|null   |1111222|null   |null   |null |text |null |null   
1234|null   |null   |null   |1111333|null |null |null |text   
1234|null   |null   |null   |1111333|null |null |null |text

expected output:

USER|ID     |Name
 
1234|1234500|text
 
1234|1234500|text 

1234|1111500|text
   
1234|1111500|text |   
1234|1111222|text |   
1234|1111222|text |   
1234|1111333|text |   
1234|1111333|text |
pyspark apache-spark-sql python
1 Answers

Try with coalesce(col1,col2,col3..coln)

Example:

df.withColumn("id",coalesce(col("ID_1"),col("ID_2"),col("ID_3"),col("ID_4"))).\
withColumn("Name",coalesce(col("Name1"),col("Name2"),col("Name3"),col("Name4"))).\
show()
user_76326950
Related