I have a worker that generates data like this:
for (const item in queue){
const data = await get_data();
data_stream.push(data);
}
There may be multiple of these workers running simultaneously, and they all push to the same stream.
This stream is Readable and is fed to a JsonStreamStringify like this:
data_stream = new stream.Readable({ objectMode: true, read() { } })
new JsonStreamStringify({foo: data_stream, ...other stuff}).pipe(some file writing stream);
When all the workers have finished, we end the stream with
data_stream.push(null)
But I get this error:
(node:17) UnhandledPromiseRejectionWarning: Error: Readable Stream has ended before it was serialized. All stream data have been lost
at JsonStreamStringify.addToStack (/usr/src/app/node_modules/json-stream-stringify/umd.js:316:32)
at JsonStreamStringify.processObject (/usr/src/app/node_modules/json-stream-stringify/umd.js:404:14)
at JsonStreamStringify.processStackTopItem (/usr/src/app/node_modules/json-stream-stringify/umd.js:506:53)
at JsonStreamStringify.__read (/usr/src/app/node_modules/json-stream-stringify/umd.js:532:21)
at Immediate.<anonymous> (/usr/src/app/node_modules/json-stream-stringify/umd.js:539:22)
at processImmediate (internal/timers.js:456:21)
Which has zero results on Google and doesn't give me a useful line number to track the bug down.
Any ideas why this error is being thrown? Or a better solution for passing data from my workers to the JsonStreamStringify object?
Desired output
The file that we write to should contain json like
{
foo: {
[data1, data2, ...]
}
}
When the program crashes, we have json like:
{
foo: {
[