Node.Js HTTP response cut off

Question

I have the following function:

const https = require("https");
async function get(options) {
    return new Promise((resolve, reject) => {
        let output = '';
        const req = https.request(options, res => {
            res.setEncoding('utf8');
            res.on('data', chunk => {
                output += chunk;
            });
            res.on('end', () => {
                resolve(JSON.parse(output));
            });
        });
        
        req.on('error', err => {
            return reject(err);
        });

        req.end();
    });
}

I'm using this function to fetch a big (about 5MB in size) JSON from an HTTP server. The problem is sometimes the HTTP response gets cut off and I get unexpected end of JSON input error. So, when the end event happens, output variable contains the first N characters of JSON, where N varies, the values I've seen (1011, 8091 ...). It is worth to mention, that only when I'm making a lot of requests this issue happens, usually 1 out of 100 requests fail with this error. All requests return approximately 5MB sized JSON.

To debug this issue, I've run the script with: NODE_DEBUG='net' node my_script.js env. variable and it turned out, that just before I'm getting the aforementioned exception, the following happens:

NET 36897: _read
....
NET 36897: _read
NET 36897: destroy
NET 36897: close
NET 36897: close handle
NET 36897: _final: not yet connected
NET 36897: emit close

This sequence of logs appear only when I get the unexpected end of JSON error. For the reference, the other successful requests ended with:

NET 36897: _read
NET 36897: _final: not ended, call shutdown()

The reason I think this issue is related to Node.Js https module - I wrote the same script in python and it works, the issue is not reproducible.

Question: why am I getting cut off JSON in output ?

Node.Js HTTP response cut off

0 Answers