How do I stream JSON objects from ExpressJS?

Question

I'm trying to stream JSON objects from an ExpressJS / Node backend API to a frontend site.

I do not want to use Sockets.IO for various reasons. As I understand it, the native streaming libraries should support streaming objects, it appears that just Express is complicating this.

My frontend code seams straight forward. I use Fetch to get my target URL, get a read stream from the response object, and set that read stream to objectMode: true.

Frontend Example:

    async function () {

      let url = "myurl";

      let response = await fetch( url, {
        method: 'GET',
        mode: 'cors',
        wtihCredentials: 'include'
      }

      const reader = response.body.getReader({objectMode: true });

      // Where things are a bit ambiguous
      let x = true;
      while (x) {
        const {done, value} = reader.read()
        if (done) { break; }

        // do something with value ( I push it to an array )

      }      
}

Backend Bode Example ( fails because of I cannot change the stream to objectMode )

router.get('/', (request, response) => {

  response.writeHead(200, { 'Content-Type' : 'application/json' });

  MongoDB.connection.db.collection('myCollection').find({}).forEach( (i) => {
    response.write(i);
  }).then( () => {
    response.end()
  })
})

Now my problem is that there does not appear to be anyway to change the ExpressJS write stream to objectMode: true. To my dismay, the ExpressJS documentation doesn't even acknoledge the existence of the write() function on the response object: https://expressjs.com/en/api.html#res

How do I change this over to objectMode: true ?

conversely, I tried to work with the writeStream as a string. The problem that I run into is that when the send buffer fills up, it does it by characters, not by the object. These means that at some point invalid JSON is passed to requester.

A suggested solution that I run into often is that I could read all of the chunks on the client and assemble valid JSON. This defeats the purpose of streaming, so Im trying to find a better way.

For what I believe is the same problem, I cannot figure out how to talk directly to the write stream object from the express code so I am unable to use the native writeStream operation writable.length in order to manually check to see if there is space for the entire JSON object as a string. This is preventing me from using stringified JSON with new line terminators.

https://nodejs.org/api/stream.html#stream_writable_writablelength

https://nodejs.org/api/stream.html#stream_writable_writableobjectmode

Could someone set me straight? I am working with 100k + records in my Mongo database, I really need partical page loading to work so that the users can start picking through the data.

How do I stream JSON objects from ExpressJS?

0 Answers