I'm using pyspark and can't find the optimal way to remove the for loop in this situation:
I have a list of months which is the input of a function, e.g.:
months = ['2022-05', '2022-06']
Then this list is given as input in a function:
def f(data, months):
for i, month in enumerate(months):
# long series of operations on the individual month
if i == 0:
out = out1
else:
out.union(out1)
# other operations
return out
How could I replace that for cycle in a more optimal way? I also would like to consider ranges of months in the future.
I looked up online and saw Window.partitionBy, but it looks that it doesn't support a complex series of operations. Also, could there be a solution with a lambda function and apply()?
Thank you in advance