"Readable Stream has ended before it was serialized" in npm/jsonstreamstringify

Viewed 52

I have a worker that generates data like this:

for (const item in queue){
    const data = await get_data();
    data_stream.push(data);
}

There may be multiple of these workers running simultaneously, and they all push to the same stream. This stream is Readable and is fed to a JsonStreamStringify like this:

data_stream = new stream.Readable({ objectMode: true, read() { } })
new JsonStreamStringify({foo: data_stream, ...other stuff}).pipe(some file writing stream);

When all the workers have finished, we end the stream with

data_stream.push(null)

But I get this error:

(node:17) UnhandledPromiseRejectionWarning: Error: Readable Stream has ended before it was serialized. All stream data have been lost
    at JsonStreamStringify.addToStack (/usr/src/app/node_modules/json-stream-stringify/umd.js:316:32)
    at JsonStreamStringify.processObject (/usr/src/app/node_modules/json-stream-stringify/umd.js:404:14)
    at JsonStreamStringify.processStackTopItem (/usr/src/app/node_modules/json-stream-stringify/umd.js:506:53)
    at JsonStreamStringify.__read (/usr/src/app/node_modules/json-stream-stringify/umd.js:532:21)
    at Immediate.<anonymous> (/usr/src/app/node_modules/json-stream-stringify/umd.js:539:22)
    at processImmediate (internal/timers.js:456:21)

Which has zero results on Google and doesn't give me a useful line number to track the bug down.

Any ideas why this error is being thrown? Or a better solution for passing data from my workers to the JsonStreamStringify object?

Desired output

The file that we write to should contain json like

{
    foo: {
        [data1, data2, ...]
    }
}

When the program crashes, we have json like:

{
    foo: {
        [
0 Answers
Related