Create boolean flag based on column value containing element of a List

user_9252815 0
Viewed 1325
def myFunction(df: DataFrame): DataFrame = {
    val myList= List("a","b","c")

    df
      .withColumn("myFlag",
        if (myList.contains(df.select(col("columnName1")))) lit("true") else lit(false))
}

I want to write a function, that takes a Dataframe, and adds a column to it, named "myFlag".

I want "myFlag" to be true if the corresponding "columnName1" has a value that is an element of "myList", false otherwise.

For simplicity, "columnName1" values and "myList" only contain Strings.

My function above will not work. Any suggestions?

apache-spark conditional-statements dataframe dataset scala
1 Answers

This can be done using isin which is defined on Column:

import spark.implicits._

df
  .withColumn("myFlag",$"columnName1".isin(myList:_*))
user_11385230
Related