pyspark dataframe iterate over the list

user_13384943 0
Viewed 65

I have a list and pyspark dataframe like below. Today my list has only 3 elements and tomorrow it might have 5 elements and the list is dynamic not static.

         my_list = ['4587','9920408','9920316']

         a=spark.createDataFrame([(101,'~1~20448~3~22901~12214~27681~9920408~20013~19957~19993~ ~ ~ ~ ~ ~'),(102, '~1~20448~4462~4586~24739~4587~9914381~9921471~12777~ ~ ~ ~ ~ ~ ~'),(103,'~1~20448~3~22901~3891~4148~9920948~14845~4230~ ~ ~ ~ ~ ~ ~'),(104, '~1~20448~3~22901~3891~4148~9920316~4211~4212~ ~ ~ ~ ~ ~ ~')], ['ID', 'MSH'])

Now i want to create with column like ind if msh like '%~4587~%' or msh like '%~9920408~%' or msh like '%~9920316~%' then it's 1 otherwise 0.

I tried like below and it's working.

   b=a.withColumn('ind', F.expr("if((msh like '%~4587~%' or msh like '%~9920408~%' or msh like '%~9920316~%'),1,0)"))

Is there a way to have a dynamically create the if condition to have msh like n times if we have n items in the list.

Appreciate your support.

pyspark apache-spark-sql
1 Answers

Multiple likes should consider using rlike regular matching, and use | to match multiple patterns at the same time.

reg_expr = '|'.join([f'~{i}~' for i in my_list])
b = a.withColumn('ind', F.expr(fr"if(msh rlike '{reg_expr}', 1, 0)"))
b.show(truncate=False)
user_170214290
Related