How to validate only specific schema from a dataframe

user_19251495 0
Viewed 39

I am trying to test specific schemas from a dataframe. In the data frame, there are some columns that have no value or null value.

     def test_column_names(self):
            schema=StructType([StructField("column 1", StringType(), True),
            StructField("column 2", StringType(), True),
            StructField("column 3", StringType(), True),
            StructField("column 4", StringType(), True),

        assert schema in df.columns``


The df.columns has more than 1000 schemas. I wanted to verify only the above schemas exist in the df.columns and assert that. I have tried ``assert schema in df.columns``. But getting AssertionError: unit test failures:
databricks apache-spark-sql testing python
1 Answers

Not a very cool implementation, still does the job :)

%python

for sccols in schema:
    for dfcols in df.schema.fields:
        if (sccols.name == dfcols.name):
            if (sccols.dataType == dfcols.dataType):
                assert True
            else:
                assert False
        else:
            pass
user_110865150
Related