Spark 2.4 drop columns based on value scala

Admin 1
Viewed 103

I am using spark 2.4 with scala 2.11 ,I want to drop column from df having value "Default" and zero.

+-------+--------+-----------+-------+
|A      |B       |C          | D     |
+-------+--------+-----------+-------+
|     1 |      1 |   Deafult |  0    |   
+-------+--------+-----------+-------+

This is what I'm expecting as an output:

+-------+--------+
|A      |B       |
+-------+--------+
|     1 |      1 |  
+-------+--------+
apache-spark dataframe filter scala
1 Answers

With your example, which has only one record in the dataframe:

// Get values as an array, mapping to true/false depending on whether it's the default
val dfValues = df.head.toSeq.map(v => v == 0 || v == "Default")

// Get column names that are not default
val columns = (df.columns zip dfValues)
                .collect { case (col, isDefault) if !isDefault => col }
                .map(col)

// Read selected columns
df.select(columns:_*)

If you have more records in your df you need to specify behavior first (should a column be included if there any value is non-default? All values non-default?) and run some aggregation.

user_28900930
Related