Count elements separated by commas from a list and save to another column

user_14133708 0
Viewed 366

I have a pyspark dataframe that looks like this:

+-------------+
|list|
+-------------+
|1,1,1,1   
+-------------+
|New,Upgrade,Old
+-------------+

How can I generate a field that counts the elements separated by commas? The ideal dataframe looks like this:

 +---------------+-----------
 |list            |count
 +----------------+----------
 |1,1,1,1         |  4
 +----------------+--------
 |New,Upgrade,Old |  3
 +----------------+-------
pyspark dataframe python
1 Answers

Use split and size functions.

Example:

df=spark.createDataFrame([('1,1,1,1',),('New,Upgrade,Old',)],['list'])
df.show()
#+---------------+
#|           list|
#+---------------+
#|        1,1,1,1|
#|New,Upgrade,Old|
#+---------------+

from pyspark.sql.functions import *

df.withColumn("count",size(split(col("list"),","))).show()
#+---------------+-----+
#|           list|count|
#+---------------+-----+
#|        1,1,1,1|    4|
#|New,Upgrade,Old|    3|
#+---------------+-----+
user_76326950
Related