I have my metadata that look like this example:
{
"metadata_id":"2",
"flow_properties": {
"flow_name_index" : 3,
"flow_name_separator" : "_"
},
"flows": [
{
"flow_name" : "CONTACT",
"file_properties": {
"separator":"¤",
"encoding-type" : "utf-8"
},
"nb_lines": 3,
"schema" : {
"fields": [
{
"name": "id_contact",
"type": "string"
},
{
"name": "first_name",
"type": "string"
},
{
"name": "last_name",
"type": "string"
}
]
}
},
{
"flow_name" : "SALES",
"file_properties": {
"separator":",",
"encoding-type" : "utf-8"
},
"nb_lines": 6,
"schema" : {
"fields": [
{
"name": "id_sales",
"type": "string"
},
{
"name": "description",
"type": "string"
},
{
"name": "total_price",
"type": "number"
}
]
}
}
]
}
I want to validate my files that come from those different flows against the correct schema in the metadata.
My question is: Is there a library in python that takes a json schema, a csv file and do the job?
I already tested goodtables, csvvalidator and pycsvschema. They do not seem to answer my needs, except for pycsvschema. However I cannot make it work. For the details about the pycsvschema, please check the issue I opened here.
I also tried to use pyspark, but I don't know if that is a good idea if my files are not that big!
Any usefull information would be appreciated!