PySpark evaluate formula stored in a DataSet column, using values from another DataSet

Viewed 223

I have a Dataset (main) storing a formula in a column:

ID | Formula
1    (a/(b+c))*100
2    m/n*100
3    d

And another Dataset (val) containing the values:

ID  |  Value
a      4
b      3
c      8
d      7
m      2
n      5

I need to evaluate the formula using the values from the val dataset, and save the result into a new main column.

I did this by iterating line by line through the main Dataset, create a variable storing a list with all the formula parameters, replace the parameter with values and evaluate. Then I used .withColumn to add the result to the main Dataset. But this is not optimal at all for large amounts of data.

Could you please help me with a hint of how can I do this without iterating line by line ?

Thank you!

0 Answers
Related