NodeJS Download and Upload In-Memory Stream Asynchronously

Question

Question
How do you asynchronously download and (resumable/multi-part) upload large MP4 files purely through Streams in NodeJS (without using filesystem)?

Scenario
I'm writing a class file that is concerned with streaming downloads from a URL to Google Drive or Dropbox purely through memory.

**The file size is known prior to the download and upload.

Visual

==============
      ^(5%)  ^(10%)
Downloader

      ==============
            ^(5%)  ^(10%)
       Uploader

Requirements

Transfer through Memory Streams (without storing anything on the filesystem)
The uploads should be multi-part resumable and may range anywhere from 5 Mb to > 20 Gb.
If a download or upload fails, there should be a set number of retries.

Curiosities

Is a duplex stream (e.g. passthrough) the correct way of doing this?
How do you convey Content-Length within an asynchronous method as visualized above?

Pseudo Code

const axios = require('axios');
const stream = require('stream');
const passtrough = new stream.PassThrough();

let sample = VideoAPI.get() // pass id
//sample.url // url located here
//sample.size // size is known prior to download or upload
//sample.contentType // content-type is known prior to download

//Download sample via Axios
axios.get(sample.url, {
    responseType: "stream"
}).then((response) => {
    //TODO: Pipe to Google Drive
    console.log('response', response)
}).catch((error) => {
    console.error(error)
})

Research

user_1743997 · Answer

I think you need sth like this :

request.get(sourceUrl).pipe(request.post(targetUrl))

In this scheme, the data will flow from sourceUrl to targetUrl but will not need to be saved in a temporary file on the server.

For clarification visit request#streaming

user_6138124 · Answer

const formData = new FormData();

axios.get(sample.url, {
    responseType: "stream"
}).then((response) => {
    //TODO: Pipe to Google Drive
    // you can directly put the response stream onto the formData object.
    formData.append('files', response.data);
    return axios.post(googleURL,{headers:{...formData.getheaders()}, data: formData});
.then((response)=>{
//response from google
console.log(response.data);
})
}).catch((error) => {
    console.error(error)
})

//N.B. The response.data steam will be appended to the formData object, this is all done in memory. Make sure the responseType is set to stream.

NodeJS Download and Upload In-Memory Stream Asynchronously

2 Answers