I have a Dataset (main) storing a formula in a column:
ID | Formula
1 (a/(b+c))*100
2 m/n*100
3 d
And another Dataset (val) containing the values:
ID | Value
a 4
b 3
c 8
d 7
m 2
n 5
I need to evaluate the formula using the values from the val dataset, and save the result into a new main column.
I did this by iterating line by line through the main Dataset, create a variable storing a list with all the formula parameters, replace the parameter with values and evaluate. Then I used .withColumn to add the result to the main Dataset. But this is not optimal at all for large amounts of data.
Could you please help me with a hint of how can I do this without iterating line by line ?
Thank you!