I am working on a dataset whose format is as follows:
+-------+----------------------------------------------------------------+--------------+
|id |expression |gender |
+-------+----------------------------------------------------------------+--------------+
|1 |CASE WHEN gender ='1' THEN true ELSE false END |1 |
|2 |CASE WHEN gender ='1' THEN true ELSE false END |1 |
|3 |CASE WHEN gender ='1' THEN true ELSE false END |2 |
|4 |CASE WHEN gender ='1' AND date<> null THEN true ELSE false END |2 |
...
+-------+----------------------------------------------------------------+--------------+
I want to apply SQL rules stored in columns of this same dataset, to check conditions and create a column and get the following dataset:
+-------+----------------------------------------------------------------+--------------+---------+
|id |expression |gender |isValid |
+-------+----------------------------------------------------------------+--------------+---------+
|1 |CASE WHEN gender ='1' THEN true ELSE false END |1 |true |
|2 |CASE WHEN gender ='1' THEN true ELSE false END |1 |true |
|3 |CASE WHEN gender ='1' THEN true ELSE false END |2 |false |
|4 |CASE WHEN gender ='1' AND date<> null THEN true ELSE false END |2 |false |
...
+-------+----------------------------------------------------------------+--------------+---------+
I am currently using Java spark and I was planning to use the following code:
myDataset = myDataset.withColumn("isValid", functions.expr("CASE WHEN gender ='1' THEN true ELSE false END");
However, the withColumn function only accepts strings and I don't know how to get the contents of the "expression" column. So this code does not work
myDataset = myDataset.withColumn("isValid", functions.expr(myDataset.col("expression");
In addition, the checks in the expression column do not always check the same conditions (it can be a check in one or more columns of the dataset, with equals, different, more than ....). After much research, I don't find anythings for solve this problem. Do you have any idea ? Thank you.