passing column name to dictionary

Viewed 59

I am facing challenges due to lack of much knowledge in spark. i have below requirement where i need to pass dataframe column name in dictionary as a key.

I am doing below steps but getting an error

Defining dict as below

promo_sub_cat_dict = {
        "Negative-Low Baseline-Low Uplift":"STOP",
        "Negative-High Baseline-Low Uplift":"SAVE0",
        "Positive-Low Baseline-High Uplift":"SAVE1",
        "Negative-High Baseline-High Uplift":"SAVE2",
        "Negative-Low Baseline-High Uplift":"SAVE3",
        "Positive-Low Baseline-Low Uplift":"KEEP1",
        "Positive-High Baseline-Low Uplift":"KEEP2",
        "Positive-High Baseline-High Uplift":"INVEST"
    }

adding column name in dataframe as below

df = waterfallmatrix.withColumn("sub_promo_cat",promo_sub_cat_dict[col("concat_flag")])
TypeError: unhashable type: 'Column'

and getting below error, can somebody help me to find the correct syntax

0 Answers
Related