when and otherwise in pyspark using independent conditions

user_16431975 0
Viewed 49

I am trying to make a new column based on list of items. cond is a separate independent list outside dataframe "ABC". A new column needs to be created based on few conditions, please see a dummy condition below:-

Rules = ['2'. '5', '18', '33', '47']

ABC = ABC.withColumn("XYZ", when ('2' in cond), col("sqr_4231"))\
                                   .when(('3' in cond), col("sqr_4999"))

but this is giving error - have tried lit('2') and lit('3') but still giving error.

pyspark apache-spark python
1 Answers

To verify if a value contains in an array you need to use:

lit('value').isin(array_value)

aggregate = aggregate.withColumn("ENR", when(lit('2').isin(Rules), col("sqr_4231"))\
                                   .when(lit('3').isin(Rules), col("sqr_4999"))
user_60802760
Related