How to convert GUID into integer in pyspark

Viewed 173

Hi Stackoverflow fams:

I am new to pyspark and trying to learn as much as I can. But for now, I want to convert GUID's into integers in pysprak. I can currently run the following statement in SQL to convert GUID's into an int.

CHECKSUM(HASHBYTES('sha2_512',GUID)) AS int_value_wanted

I wanted to do the same thing in pyspark and tried to create a temporary table out of spark dataframe and add the above statement in the sql query. But the code keeps throwing "Undefined function: 'CHECKSUM'". Is there a way I can add the "CHECKSUM" function into pyspark or do the same thing using another pyspark way?

from awsglue.context import GlueContext
from pyspark.sql import SQLContext

glueContext = GlueContext(SparkContext.getOrCreate())
spark_session = glueContext.spark_session
sqlContext = SQLContext(spark_session.sparkContext, spark_session)

spark_df =  spark.createDataFrame(
    [("2540f487-7a29-400a-98a0-c03902e67f73", "1386172469"),
    ("0b32389a-ce01-4e6a-855c-15940cc91e9e", "-2013240275")],
    ("GUDI","int_value_wanted")
)

spark_df.show(truncate=False)
spark_df.registerTempTable('temp')
new_df = sqlContext.sql("SELECT .*, CHECKSUM(HASHBYTES('sha2_512', GUDI)) AS detail_id FROM temp")
new_df.show(truncate=False)
+------------------------------------+----------------+
|GUDI                                |int_value_wanted|
+------------------------------------+----------------+
|2540f487-7a29-400a-98a0-c03902e67f73|1386172469      |
|0b32389a-ce01-4e6a-855c-15940cc91e9e|-2013240275     |
+------------------------------------+----------------+

Thanks

1 Answers

There is a sha2 built-in function, which returns the checksum for the SHA-2 family as a hex string. SHA-512 is also supported.

Related