Add column conditionnally in Spark dataframe

Viewed 24

I would like to know if there is a way to add a column to a Spark dataframe in the middle of a chain conditionnally?

I would like to do something like that:

val newDf = df
  .long
  .chain
  .of
  .instructions
  .withColumn("newColumn", operation(col("anotherColumn"))) if "anotherColumn" exists otherwise don't add a column
  .another
  .long
  .chain
  .of
  .instructions

I know that I could break my chain like that:

val dfTmp = df.long
  .chain
  .of
  .instructions

val dfTmp2 = if (dfTmp.contains("anotherColumn"))
  dfTmp.withColumn("newColumn", operation(col("anotherColumn")))
else
  dfTmp

val newDf = dfTmp2
  .long
  .chain
  .of
  .instructions

But it affects code readability quite a lot as it creates useless intermediate variables. I would like an easy-to-read way to include somehow the test in the chain.

0 Answers
Related