How to merge the value of several columns into a map in Spark Dataframe

As I understand what you are trying is to have columns "temperature", "humidity", "brightness" together into a single map.

You can try this:

// First you create a sequence with those columns
val columnsToMap = Seq("temperature", "humidity", "brightness")

// Then you write a udf to merge maps
def mergeMapUdf= udf((map1: Map[String, Int], map2: Map[String, Int])=> map1 ++ map2)

// Finally you merge them as a map, one by one
val sensorsDf2 = columnsToMap.foldLeft(sensorsDf.withColumn("measures",
 lit(null)))((df,column) => df.withColumn("measures",
 when(col("measures").isNull,map(lit(column),col(column)))
.otherwise(mergeMapUdf( col("measures"), map(lit(column), col(column)))))
.drop(col(column)))

FoldLeft: What is does is to iterate through the list of columns columnsToMap and operates with each of the values. The operation in this case is to add or merge a new map to the column "measures".

The result would be as follows

+-------+----------------------------------------------------------+
|sensor |measures                                                  |
+-------+----------------------------------------------------------+
|sensor1|Map(temperature -> 26, humidity -> 30, brightness -> 1000)|
|sensor2|Map(temperature -> 25, humidity -> 30, brightness -> 1100)|
|sensor3|Map(temperature -> 26, humidity -> 30, brightness -> 1000)|
+-------+----------------------------------------------------------+

I am not sure if you want to drop the original columns. In case you want to keep them, remove the line .drop(col(column))

Less lines of code

import org.apache.spark.sql.functions._

object ColToMap {

  def main(args: Array[String]): Unit = {

    val spark = Constant.getSparkSess

    import spark.implicits._

    val sensorsDf = Seq(
      ("sensor1", 26, 30, 1000),
      ("sensor2", 25, 30, 1100),
      ("sensor3", 26, 30, 1000)
    ).toDF("sensor", "temperature", "humidity", "brightness")

//Create a List of columns ready to be mapped
    val colToMap = List("temperature", "humidity", "brightness")
  .flatMap(colName =>  List(lit(colName) , col(colName) ))

    sensorsDf
      .withColumn("measures", map(colToMap: _ *)).show(false)
  }

}

//Output
+-------+-----------+--------+----------+-------------------------------------------------------+
|sensor |temperature|humidity|brightness|measures                                               |
+-------+-----------+--------+----------+-------------------------------------------------------+
|sensor1|26         |30      |1000      |[temperature -> 26, humidity -> 30, brightness -> 1000]|
|sensor2|25         |30      |1100      |[temperature -> 25, humidity -> 30, brightness -> 1100]|
|sensor3|26         |30      |1000      |[temperature -> 26, humidity -> 30, brightness -> 1000]|
+-------+-----------+--------+----------+-------------------------------------------------------+