How to read large JSON files that have different data types in it

Viewed 134

I am creating an npm package that will read json files and then validate the content against some predefined json-schemas everything was working fine when I was testing against small size (~1MB) files but when I started reading 50MB and larger sizes things started to fail, like when I reach the code to pares the file Allocation failed - JavaScript heap out of memory is being thrown so I tried to increase the size of node memory node --max-old-space-size=4096 but now the parsing is taking forever (waited almost one hour and nothing happened).

So here is how the json file may look like:

[{
    "numberVerification": [
        {
            "longNumber": 281474976710655
        }
    ]
},
{
    "metaData": [
        {
            "name": "nodes",
            "elementCount": 155,
            "idCounter": 155,
            "version": "1.0",
        },
        {
            "name": "edges",
            "elementCount": 312,
            "idCounter": 312,
            "version": "1.0",
        },
        {
            "name": "networkAttributes",
            "elementCount": 14,
            "idCounter": 14,
            "version": "1.0",
        },
        {
            "name": "nodeAttributes",
            "elementCount": 330,
            "idCounter": 330,
            "version": "1.0",
        },
        {
            "name": "edgeAttributes",
            "elementCount": 3120,
            "idCounter": 3120,
            "version": "1.0",
        },
        {
            "name": "cartesianLayout",
            "elementCount": 155,
            "idCounter": 156
        },
    ]
},
{
    "nodes": [
        {
            "@id": 0,
            "n": "TYK2",
            "r": "uniprot:P29597"
        },
        {
            "@id": 1,
            "n": "ISGF3 complex",
            "r": "signor:SIGNOR-C124"
        },
        {...}
    ]
},
{
    "edges": [
        {
            "@id": 0,
            "s": 0,
            "t": 1,
            "i": "up-regulates activity"
        },
        {
            "@id": 1,
            "s": 2,
            "t": 1,
            "i": "up-regulates activity"
        },
        {...}
    ]
},
{
    "nodeAttributes": [
        {
            "po": 0,
            "n": "type",
            "v": "protein"
        },
        {
            "po": 0,
            "n": "location",
            "v": "cytoplasm"
        },
        {...}
    ]
},
{
    "edgeAttributes": [
        {
            "po": 0,
            "n": "citation",
            "v": [
                "pubmed:15120645"
            ],
            "d": "list_of_string"
        },
        {
            "po": 0,
            "n": "mechanism",
            "v": "phosphorylation"
        },
        {...}
    ]
},
{
    "cartesianLayout": [
    {
        "node": 0,
        "x": 97.73626669665249,
        "y": -114.99468800778627
    },
    {
        "node": 1,
        "x": 307.72737757573987,
        "y": 4.091777979752425
    },
    {...}
]
},
{
    "status": [
        {
            "error": "",
            "success": true
        }
    ]
}]

As you can see each object of the main array is of different type.

I was using fs.readFileSync to read json files and then I parse the whole file using JSON.parse(), but then I find that in order to read big json files I need to use streams but now how I am supposed to validate against json-schema? also I am doing some additional custom validation on the data, like checking if the @id property is unique, validate that the node (in cartesian layout array), t & s (in edges array) properties are pointing on a real @id in the nodes array, and after that I want to make some further statistics on the data.

So is there a way to read the large json files and parse it, and if it is applicable, how can I keep the validation running? and the statistic part do I need to use any external providers?

The json files are generated from a 3rd party so I can't really do anything with the structure of the data and how it is presented inside of them.

If anyone want me to provide any additional things let me know, I appreciate the help.

1 Answers

To work with JSON objects within streams, you can use JSONStream npm library (see docs for usage). You will have to use JSONPath syntax to reach desired nodes (including the root node, if necessary) and perform a "rolling" validation according to your rules.

Related