I have a spark dataframe which I am using to store keys generated for encryption. I use a UDF to generate a key for each row but whenever the dataframe is queried the keys change. I have tried wrapping the function in lit() but it still changes. How can I make this column Immutable?
Code:
def generate_key():
encryptionKey = Fernet.generate_key().decode('utf-8')
return encryptionKey
get_key = udf(generate_key, StringType())
def key_table(df):
df = df.select("id").withColumn('encKey', lit(get_key()))
return df