Case when statement to conditionally populate columns in pyspark

Viewed 65

I have a PySpark as show below:

df6 = df_n_a \
        .withColumn("x", explode_outer(col("x"))) \
        .withColumn("y", when( \
            col('x.e')=='S',  lit(None).cast("string")) \
                    .when(col('x.e')=='C', element_at("m.n.o", -1))) \
            .select(
                from_unixtime(col("x.w")).alias("w"),
                col("x.e").alias("e"),
                col("y"),
        )

I want to populate the column 'y' as NULL if x.e is 'S', and if it is 'C' then populate it with a logic using element_at since it is an array. The problem here is that when execution begins, even when x.e is 'S' the 'when' for the input as 'C' gets executed. This is producing a key missing error, as records of type 'x.e'=='S' in input JSON won't have the keys m.n.o. Is there a way to strictly impose a logic wherein, if x.e=='S' then make it NULL and not to continue with the next when statement. I've tried with otherwise after the first when and it still won't work. Thanks!

0 Answers
Related