I would like to know if there is a way to add a column to a Spark dataframe in the middle of a chain conditionnally?
I would like to do something like that:
val newDf = df
.long
.chain
.of
.instructions
.withColumn("newColumn", operation(col("anotherColumn"))) if "anotherColumn" exists otherwise don't add a column
.another
.long
.chain
.of
.instructions
I know that I could break my chain like that:
val dfTmp = df.long
.chain
.of
.instructions
val dfTmp2 = if (dfTmp.contains("anotherColumn"))
dfTmp.withColumn("newColumn", operation(col("anotherColumn")))
else
dfTmp
val newDf = dfTmp2
.long
.chain
.of
.instructions
But it affects code readability quite a lot as it creates useless intermediate variables. I would like an easy-to-read way to include somehow the test in the chain.