Is there a way to make a child process output stream flush its data more frequently?

user_741970 0
Viewed 1167

I am trying to pipe a child process' standard output to the parent's standard output:

import {exec} from 'child_process';

console.log(new Date() + " starting")
const child = exec(/* some command */);
child.stdout.pipe(process.stdout);

This works, but the child process generates data quite slowly relative to the size of the pipe's buffer. The data comes in large chunks and not frequently.

For example, if I watch the child output stream like this:

child.stdout.on('data', data => console.log(new Date(), data.length));

The output is

2017-11-15T21:53:44.128Z starting
2017-11-15T21:53:58.319Z 8192
2017-11-15T21:54:02.321Z 8192
2017-11-15T21:54:07.384Z 8192
2017-11-15T21:54:11.333Z 8192
2017-11-15T21:54:15.281Z 8192
2017-11-15T21:54:19.008Z 3967

Is there a way to have the child output stream use a smaller buffer or flush more frequently?

node.js
1 Answers

The OS and child process are in control of output buffering.

As an example, Python has a -u option that causes writes to be flushed (also the PYTHONUNBUFFERED env var). The following example adapted from the question shows the difference in behaviour of a command that writes an integer to screen every second when using unbuffered and normal output.

const {exec} = require('child_process')

function run(cmd){
  return new Promise((resolve, reject) => {
    console.log("%s starting %s", Date.now(), cmd)
    const child = exec(cmd)
    child.stdout.pipe(process.stdout)
    child.stderr.pipe(process.stderr)
    child.on('exit', exit => {
      console.log('%s exit', Date.now(), exit)
      if ( exit === 0 ) return resolve(exit)
      reject(new Error(exit))
    })
  })
}

async function go(){
  await run('python -uc "import time; [print(i,str(time.sleep(1))) for i in range(10)]"')
  await run('python -c "import time; [print(i,str(time.sleep(1))) for i in range(10)]"')
}

go()

If the child process being run doesn't have an equivalent option or config to flush output then this Unix + Linux question includes a number of tricks to disable buffered output by modifying how the child process is run, by either allocating a pseudo terminal or modifying it's buffers directly.

There's also node-pty and node-pty2 which will spawn processes with a pseudo terminal which would be similar to what expects unbuffer command does. I've not used either module before though so can't vouch for them.

user_13186940
Related