I am facing challenges due to lack of much knowledge in spark. i have below requirement where i need to pass dataframe column name in dictionary as a key.
I am doing below steps but getting an error
Defining dict as below
promo_sub_cat_dict = {
"Negative-Low Baseline-Low Uplift":"STOP",
"Negative-High Baseline-Low Uplift":"SAVE0",
"Positive-Low Baseline-High Uplift":"SAVE1",
"Negative-High Baseline-High Uplift":"SAVE2",
"Negative-Low Baseline-High Uplift":"SAVE3",
"Positive-Low Baseline-Low Uplift":"KEEP1",
"Positive-High Baseline-Low Uplift":"KEEP2",
"Positive-High Baseline-High Uplift":"INVEST"
}
adding column name in dataframe as below
df = waterfallmatrix.withColumn("sub_promo_cat",promo_sub_cat_dict[col("concat_flag")])
TypeError: unhashable type: 'Column'
and getting below error, can somebody help me to find the correct syntax