Using Spark DataFrame filter with a List of column names

Question

I have to filter non-null column values in a Spark DataFrame using a List[String]:

val keyList = List("columnA", "columnB", "columnC", "columnD", ...)

For a single column named key, the syntax should be :

val nonNullDf = df.filter(col("key").isNotNull)

My question is how to use the keyList into the previous filter?

user_14165730 · Answer

You can generate a filter by doing a map-reduce on keyList.

Use and if you want to keep the rows where all columns are not null, or use or if you want to keep the rows where any column is not null.

val nonNullDf = df.filter(keyList.map(col(_).isNotNull).reduce(_ and _))