Pyspark conditional function evaluation based on another column

Viewed 156

I have a sample data set like below

sample_data = [('A', 'Chetna', 5, 'date_add(date_format(current_date(), \'yyyy-MM-dd\'), 7)'), 
           ('B', 'Tanmay', 6, '`date_add(date_format(current_date(), \'yyyy-MM-dd\'), 1)`'),
           ('C', 'CC', 2, '`date_add(date_format(current_date(), \'yyyy-MM-dd\'), 3)`'), 
           ('D', 'TC', 9, '`date_add(date_format(current_date(), \'yyyy-MM-dd\'), 5)`')]
df = spark.createDataFrame(sample_data, ['id', 'name', 'days', 'applyMe'])

from pyspark.sql.functions import lit
df = df.withColumn("salary", lit('days * 60'))

I am trying to evaluate the function provided in applyMe column and salary. So far have tried doing it with expr and eval but no luck. Could someone please point me in right direction to achieve the desired output.

0 Answers
Related