Given a table with two columns: DEVICEID and DEVICETYPE
How can I update column DEVICETYPE if the string length in DEVICEID is 5:
from pyspark.sql.functions import *
df.where(length(col("DEVICEID")) = 5).show()
PYSPARK: how can I update a value in a column based in a condition
Viewed 1310
1 Answers
Use when+otherwise statement and check the length of deviceid==5 update new value.
Example:
df=spark.createDataFrame([('abcde',1),('abc',2)],["DEVICEID","DEVICETYPE"])
from pyspark.sql.functions import *
df.withColumn("new_col",when(length(col("deviceid")) ==5,lit("new_length")).otherwise(col("DEVICEID"))).show()
#+--------+----------+----------+
#|DEVICEID|DEVICETYPE| new_col|
#+--------+----------+----------+
#| abcde| 1|new_length|
#| abc| 2| abc|
#+--------+----------+----------+
Related