Great Expectation and Spark : get full list of include_unexpected_rows

user_6609482 0
Viewed 138

I'm currently testing my datasets and so far so good, unfortunately i'm unable to get rows that don't match my expectations. I'm using a SparkDFExecutionEngine Execution Engine

For exemple :

validator.expect_column_values_to_not_be_null(column="EmailAddress",result_format="COMPLETE",include_unexpected_rows=True)

does not yield the rest of the row when the email address is null, which means I have to manually read the dataset again by filtering on null. Is there a good way with GE to get the row when the expectation is not matched ? Thank you

great-expectations pyspark databricks python-3.x
1 Answers

This appears to be a bug, specifically with the expect_column_values_to_not_be_null Expectation. If you file a Github Issue, the team should be able to resolve it. Then this should work like all of the other Expectations.

user_101611900
Related