nodejs http. large file transfer fails with 'ERR_STREAM_PREMATURE_CLOSE'

user_13741041 0
Viewed 354

Im trying to send 10 GB data from a node client to a node http server. It works fine until i reach about 5 GB. Protocol is http.

It works fine until i reach about 5 GB. Error I get on server side is [ERR_STREAM_PREMATURE_CLOSE] and on client side RequestError: read ECONNRESET.

example server:

const http = require('http')
const fs = require('fs')
const util = require('util')
const stream = require('stream')

const pipeline = util.promisify(stream.pipeline)
const host = 'localhost'
const port = 4000

const server = http.createServer(async (req, res) => {
  try {
    const writeStream = fs.createWriteStream('file', { encoding: 'binary' })
    await pipeline(req, writeStream)
  } catch (e) {
    console.log(e)
  }
})

server.listen(port, host, () => {
  console.log(`Server is running on http://${host}:${port}`)
})

example client

const got = require('got')
async function upload () {
  await pipeline(fs.createReadStream(filename), got.stream.post('http://localhost:4000/upload'))
}

I have tried with different servers (raw http and express) and clients (raw node, request, node fetch). Ive also tried multipart with busboy. Same problem.

Im trying this running node v14.4.0 on a Mac.

node.js http
1 Answers

I tried transferring a 10 gb file on my computer and it worked for me.

Server example:

const http = require('http');
const fs = require('fs');

const PORT = 8333;

const server = http.createServer((req, res) => {
    const writeStream = fs.createWriteStream('./result.txt');
    req.pipe(writeStream);
    req.on('end', () => {
        console.log('The file was successfully written.');
        res.end('OK');
    });
});

server.listen(PORT, () => {
    console.log(`Server has been started on port ${server.address().port}`);
});

Client example

const http = require('http');
const fs = require('fs');

function upload() {
    const readStream = fs.createReadStream('./test.txt');
    const request = http.request('http://localhost:8333', { method: 'POST' }, (res) => console.log(`STATUS: ${res.statusCode}`));
    readStream.pipe(request);
    readStream.on('end', () => request.end());
}

upload();

I am not saying that this code is very correct, but you can try it, since it worked for me.

I ran this example on Node.js v12.4.1 and Windows 10.

user_97662880
Related