Convert PySpark dataframe column from list to string

user_2228884 0
Viewed 27324

I have this PySpark dataframe

+-----------+--------------------+
|uuid       |   test_123         |    
+-----------+--------------------+
|      1    |[test, test2, test3]|
|      2    |[test4, test, test6]|
|      3    |[test6, test9, t55o]|

and I want to convert the column test_123 to be like this:

+-----------+--------------------+
|uuid       |   test_123         |    
+-----------+--------------------+
|      1    |"test,test2,test3"  |
|      2    |"test4,test,test6"  |
|      3    |"test6,test9,t55o"  |

so from list to be string.

how can I do it with PySpark?

pyspark apache-spark-sql apache-spark python
3 Answers

As of version 2.4.0, you can use array_join.Spark docs


from pyspark.sql.functions import array_join

df.withColumn("test_123", array_join("test_123", ",")).show()
user_55112650
Related