I am creating an npm package that will read json files and then validate the content against some predefined json-schemas everything was working fine when I was testing against small size (~1MB) files but when I started reading 50MB and larger sizes things started to fail,
like when I reach the code to pares the file Allocation failed - JavaScript heap out of memory is being thrown so I tried to increase the size of node memory node --max-old-space-size=4096 but now the parsing is taking forever (waited almost one hour and nothing happened).
So here is how the json file may look like:
[{
"numberVerification": [
{
"longNumber": 281474976710655
}
]
},
{
"metaData": [
{
"name": "nodes",
"elementCount": 155,
"idCounter": 155,
"version": "1.0",
},
{
"name": "edges",
"elementCount": 312,
"idCounter": 312,
"version": "1.0",
},
{
"name": "networkAttributes",
"elementCount": 14,
"idCounter": 14,
"version": "1.0",
},
{
"name": "nodeAttributes",
"elementCount": 330,
"idCounter": 330,
"version": "1.0",
},
{
"name": "edgeAttributes",
"elementCount": 3120,
"idCounter": 3120,
"version": "1.0",
},
{
"name": "cartesianLayout",
"elementCount": 155,
"idCounter": 156
},
]
},
{
"nodes": [
{
"@id": 0,
"n": "TYK2",
"r": "uniprot:P29597"
},
{
"@id": 1,
"n": "ISGF3 complex",
"r": "signor:SIGNOR-C124"
},
{...}
]
},
{
"edges": [
{
"@id": 0,
"s": 0,
"t": 1,
"i": "up-regulates activity"
},
{
"@id": 1,
"s": 2,
"t": 1,
"i": "up-regulates activity"
},
{...}
]
},
{
"nodeAttributes": [
{
"po": 0,
"n": "type",
"v": "protein"
},
{
"po": 0,
"n": "location",
"v": "cytoplasm"
},
{...}
]
},
{
"edgeAttributes": [
{
"po": 0,
"n": "citation",
"v": [
"pubmed:15120645"
],
"d": "list_of_string"
},
{
"po": 0,
"n": "mechanism",
"v": "phosphorylation"
},
{...}
]
},
{
"cartesianLayout": [
{
"node": 0,
"x": 97.73626669665249,
"y": -114.99468800778627
},
{
"node": 1,
"x": 307.72737757573987,
"y": 4.091777979752425
},
{...}
]
},
{
"status": [
{
"error": "",
"success": true
}
]
}]
As you can see each object of the main array is of different type.
I was using fs.readFileSync to read json files and then I parse the whole file using JSON.parse(), but then I find that in order to read big json files I need to use streams but now how I am supposed to validate against json-schema? also I am doing some additional custom validation on the data, like checking if the @id property is unique, validate that the node (in cartesian layout array), t & s (in edges array) properties are pointing on a real @id in the nodes array, and after that I want to make some further statistics on the data.
So is there a way to read the large json files and parse it, and if it is applicable, how can I keep the validation running? and the statistic part do I need to use any external providers?
The json files are generated from a 3rd party so I can't really do anything with the structure of the data and how it is presented inside of them.
If anyone want me to provide any additional things let me know, I appreciate the help.