I am using python great expectation to validate my data using Apache spark. Basically I would like to add a flag is_valid against each row of the data frame. To add this flag, I need to apply multiple checks on each column with in the row. Great expectation is giving result for a column considering the whole rows. Is there any way I can achieve this using Great expectations?
1 Answers
If you are trying to append to the dataframe you are validating with Great Expectations, currently this is not directly possible from within Great Expectations. You could add a step to your pipeline which would parse the Validation Result produced by your Checkpoint, and then append to your dataframe based on the results of that.
Related