I am trying to create a SHA256 hash of each row in a dataframe.
import org.apache.spark.sql.functions.{col, concat, sha2}
val finalResultWithHash = finalResult.withColumn("ROWHASH", sha2(concat(finalResult.columns.map(col):_*), 256))
When I had only one column in the dataframe it seemed to be working.
Later in the code I write the dataframe as a CSV and the rowhash column is empty. I haven't been able to find any documentation on what I am doing wrong.
Thank you in advance.
Another way of doing it is by using a foldLeft():
val df2 = df.withColumn("rowsha",sha2(df.columns.foldLeft(lit(""))((x,y)=>concat(x,col(y))),256))
Folding will concat all columns left to right before hashing it:
df.withColumn("rowsha",sha2(df.columns.foldLeft(lit(""))((x,y)=>concat(x,col(y))),256)).explain()
== Physical Plan ==
*(1) Project [c1#10, c2#11, c3#12, c4#13, sha2(cast(concat(, c1#10, c2#11, c3#12, 4#13) as binary), 256) AS rowsha#165]
+- *(1) ...
However, if any of the columns in concatenation contain NULLs, the result will be also NULL. To safeguard against that you might want to use something like
val df2 = df.withColumn("rowsha",sha2(df.columns.foldLeft(lit(""))((x,y)=>concat(x,coalesce(col(y),lit("n/a"))),256))
For some reason below code works for me for multiple columns
val finalResultWithHash = personDF.withColumn("ROWHASH", sha2(concat(personDF.columns.map(col): _*), 256))
+-----+-----+---+------+--------------------+
|FName|LName|Age|Gender| ROWHASH|
+-----+-----+---+------+--------------------+
| A| B| 29| M|c4ae6946a295e9d74...|
| A| C| 12| |89a18fdc3ddb3c2fd...|
| B| D| 35| F|ef1c89dfc765c7e1e...|
| Q| D| 85| |cd91aa387a7e6a180...|
| W| R| 14| |e9ff9bb78fd93a13a...|
+-----+-----+---+------+--------------------+
Could it be just the bracket placement mistake ...