How to substitute this 'for' cycle in pyspark

Viewed 21

I'm using pyspark and can't find the optimal way to remove the for loop in this situation:

I have a list of months which is the input of a function, e.g.: months = ['2022-05', '2022-06'] Then this list is given as input in a function:

def f(data, months):
   for i, month in enumerate(months):
      # long series of operations on the individual month
      if i == 0:
         out = out1
      else:
          out.union(out1)
      # other operations
   return out

How could I replace that for cycle in a more optimal way? I also would like to consider ranges of months in the future. I looked up online and saw Window.partitionBy, but it looks that it doesn't support a complex series of operations. Also, could there be a solution with a lambda function and apply()? Thank you in advance

0 Answers
Related