I am trying to stream a relatively large data file (1 Gig) from an Angular app running in the web browser to a Node.js server running on an embedded device. Streaming is necessary because the large file cannot fit into memory on the embedded device.
I was hoping to find a library that implements the various Stream API standards:
https://developer.mozilla.org/en-US/docs/Web/API/WritableStream
https://nodejs.org/api/stream.html#class-streamreadable
So I could do something simple on the web browser side like:
source_file.stream().pipeTo(socket);
And then on the Node.JS side:
socket.pipe(destination_file);
My first instinct was to use the native browser Fetch API:
https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API
But it turns out that most browsers only support streams for responses and not requests. Unfortunately, "Streaming request body" appears to be missing from the Browser Support Matrix in the documentation so this fact was not immediately obvious. Consequently, I opened a documentation issue here: https://github.com/mdn/browser-compat-data/issues/12188. A nice article on how to do this by enabling experimental browser support can be found here: https://web.dev/fetch-upload-streaming/. As a point of interest, for HTTP/1.1, this functionality is implemented on top of the chunked transfer encoding: https://en.wikipedia.org/wiki/Chunked_transfer_encoding.
Next, I looked at libraries and found the lovely websocket-stream library here:
https://www.npmjs.com/package/websocket-stream.
While this worked nicely on the Node.JS side, the browser side quickly became a nightmare because of the need to "browserify" it to work on the browser. This sadly pulls in a ton of browserified dependencies into the Angular project and involves some complicated configuration to use them.
Finally, I gave up looking and implemented some extremely basic streaming using the above mentioned stream APIs over a standard WebSocket connection. This unfortunately introduces a level of complexity I would rather not maintain which is why I continue to look for better alternatives. I may need to use the web-streams-polyfills browser side for those browsers that don't support the streams API natively yet: https://www.npmjs.com/package/web-streams-polyfill.
Other options I considered and rejected:
- stream-http (supports stream API but relies on the browser APIs/support so streaming is likely 1 directional) https://github.com/jhiesey/stream-http
- http-browserify (also relies on browser APIs/support) https://github.com/browserify/http-browserify
- XMLHttpRequest (no stream API support) https://developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequest
- WebRTC (no stream API support + complicated connection negotiation) https://webrtc.org/
- socket.io (no stream API support) https://socket.io/
- Axios (no stream API support) https://axios-http.com/
Any suggestions of libraries or technologies would be much appreciated!
Alternatively, if I prematurely rejected any of the previous technologies and there is an easy way to use them for streaming from the web browser then I'd love to hear how to use them.
Thanks!