How to add python list to a pyspark column?

user_10795596 0
Viewed 192

I have a python list say l=["abc is good", "xyz is bad"]

I already have a pyspark dataframe with columns 'A' and 'B' which are of string datatype as shown below

A B
One Two

Now I need new column C which can holds the list l

After changes the dataframe should look like this

A B C
One Two ["abc is good", "xyz is bad"] 
sample code:
df=spark.createDataFrame([("one","two")],["A","B"])
pyspark apache-spark
1 Answers

Literal function doesn't support python list as arraytype. You need to join the list elements into string first and use that as literal value in split function in pyspark sql as follows:

from pyspark.sql import functions as F

your_list = ["abc is good", "xyz is bad"]
df = df.withColumn("C",F.split(F.lit(','.join(your_list)),","))
user_92691630
Related