I'm attempting to adjust the buffer size (highWaterMark) for the stdout from a child_process.spawn() call, and I'm having a hard time doing this successfully. Below is some code that illustrates what I'm looking for. I'm trying to perform some benchmarking with various stdout buffer sizes, but until I can actually adjust the buffer size, I won't be able to do so. Any advice would be appreciated, thanks!
// I'm using the `stdout` from a child_process.spawn() call to feed into another stream, and I'm having a hard time increasing the buffer size on stdout.
// The code below illustrates the issue:
const child_process = require('child_process');
const stream = require('stream');
const fs = require('fs');
const compressStream = child_process.spawn("cat", ["testfile_32MB"], {writableHighWaterMark: 1024*1024, highWaterMark: 1024*1024}); // <-- child_process.spawn does not have highWaterMark or writableHighWaterMark options, so including them has no effect, but I'm including it just as an illustration of what I want.
compressStream.stdout.writableLength = 1024*1024; // <-- This does not help.
compressStream.stdout._writableState.writableLength = 1024*1024; // <-- This does not help.
compressStream.stdout._writableState.writableHighWaterMark = 1024*1024; // <-- This does not help.
compressStream.stdout.writableHighWaterMark = 1024*1024; // <-- This does not help.
compressStream.stdout._readableState.highWaterMark = 1024*1024; // <-- This does not help.
compressStream.stdout._writableState.highWaterMark = 1024*1024; // <-- This does not help.
compressStream.stdout.on("data", (data) => {
console.log("data length [from spawn()]:", data.length); // <-- data.length will always be <= 64KB! How can we get it higher?
});
const readStream = fs.createReadStream("testfile_32MB", {highWaterMark: 1024*1024});
readStream.on("data", (data) => {
console.log("data length [from createReadStream()]:", data.length); // <-- This prints values <= 1MB, which is what I wish I had for child_process.spawn/stdout...
});
Note, you'll need a file called "testfile_32MB" in place to run this example code, and it should be at least a few megabytes in size.