Validate a csv file against a json schema in python

Viewed 803

I have my metadata that look like this example:

{
  "metadata_id":"2",
  "flow_properties": {
    "flow_name_index" : 3,
    "flow_name_separator" : "_"
  },
  "flows": [
    {
      "flow_name" : "CONTACT",
      "file_properties": {
        "separator":"¤",
        "encoding-type" : "utf-8"
      },
      "nb_lines": 3,
      "schema" : {
        "fields": [
        {
          "name": "id_contact",
          "type": "string"
        },
        {
          "name": "first_name",
          "type": "string"
        },
        {
          "name": "last_name",
          "type": "string"
        }    
        ]
      }
    },
    {
      "flow_name" : "SALES",
      "file_properties": {
        "separator":",",
        "encoding-type" : "utf-8"
      },
      "nb_lines": 6,
      "schema" : {
        "fields": [
        {
          "name": "id_sales",
          "type": "string"
        },
        {
          "name": "description",
          "type": "string"
        },
        {
          "name": "total_price",
          "type": "number"
        }
        ]
      }
    }
  ]
}

I want to validate my files that come from those different flows against the correct schema in the metadata.

My question is: Is there a library in python that takes a json schema, a csv file and do the job? I already tested goodtables, csvvalidator and pycsvschema. They do not seem to answer my needs, except for pycsvschema. However I cannot make it work. For the details about the pycsvschema, please check the issue I opened here.

I also tried to use pyspark, but I don't know if that is a good idea if my files are not that big!

Any usefull information would be appreciated!

0 Answers
Related