How to get column values from list which contains column names in spark scala dataframe

user_7631188 0
Viewed 723

I have a config defined which contains a list of column for each table to be used as a dedup key

for ex: config 1 :

val lst = List(section_xid, learner_xid)

these are the column that needs to be used as a dedup keys. This list is dynamic some table will have 1 value some will have 2 or 3 values in it

what I am trying to do is build a single key column from this list

df.
.withColumn( "dedup_key_sk", uuid(md5(concat($"lst(0)",$"lst(1)"))) )

how do I make this dynamic which will work for any number of columns in list . I tried doing this

df.withColumn("dedup_key_sk", concat(Seq($"col1", $"col2"):_*))

For this to work I had to convert list to Df and each value in list needs to be in separate columns I was not able to figure that out.

tried doing this but didn't work

val res = sc.parallelize(List((lst))).toDF

ANy input here will be appreciated . Thank you

apache-spark-sql apache-spark scala
1 Answers

The list of strings can be mapped to a list of columns (using functions.col). This list of columns can then be used with concat:

val lst: List[String] = List("section_xid", "learner_xid")
df.withColumn("dedup_key_sk", concat(lst.map(col):_*)).show()
user_21298010
Related