Getting TypeError in WHEN and OTHERWISE condition statements pyspark

user_16525263 0
Viewed 144

I'm not able to get past this:

with the below code Im getting

TypeError: when() missing 1 required positional argument: 'value'

windowval=(Window.partitionBy('Location').orderBy('month_in_timestamp')
               .rangeBetween(Window.unboundedPreceding, 0))

df1 = df.withColumn('NEW_COL', F.when(col('col1'). \
                                     isin('data1', \
                                          'data2', \
                                          'data3', \
                                          'data4',F.sum('col2').over(windowval))) \
                                          ,F.when(col('col1'). \
                                     isin('data5', \
                                          'data6', \
                                          'data7', F.avg('col2').over(windowval))))

I tried this as well, and Im getting

java.lang.IllegalArgumentException: otherwise() can only be applied on a Column previously generated by when()

df_actual_ytd = df_tgt_reach.withColumn('NEW_COL', F.when(col('col1'). \
                                     isin('data1', \
                                          'data2', \
                                          'data3', \
                                          'data4',F.sum('col2').over(windowval))) \
                                          .otherwise(F.avg('col2').over(windowval)))
pyspark apache-spark python
1 Answers

I tried to rework your indentation to understand your when clause, add made some correction s that seem necessary. I cannot test the updated code as I do not have the context:

df1 = df \
    .withColumn('NEW_COL',
                F.when(F.col('col1').isin('data1', 'data2', 'data3', 'data4', F.sum('col2').over(windowval)),
                        F.when(F.col('col1').isin('data5', 'data6', 'data7'), F.avg('col2').over(windowval))
                        )
                )

I'm not sure you wanted to have a new when() clause if your first condition is matched, but anyway the 2nd when() did not have the value if the requirements were met. According to another topic, here is an alternative:

df1 = df \
    .withColumn('NEW_COL',
                F.when(F.col('col1').isin('data1', 'data2', 'data3', 'data4'), F.sum('col2').over(windowval))
                .when(F.col('col1').isin('data5', 'data6', 'data7'), F.avg('col2').over(windowval))
                )

If this is what you want, you might also add an .otherwise() statement at one point.

user_108916750
Related