Go exec.Command do not return output to stdout when I use cmd.Process.wait()

Question

Go version: go version go1.18 linux/arm64

What I expected: The program run script and return output result in each run.

What happen: The program run and sometimes does not return output.

I intentionally use cmd.Process.wait() because the code below is part of my application which needs to run script "service mysql start". If I used cmd.Wait(), my application would wait for mysql to finish and that is not what I want, so I use cmd.process.wait.

package main

import (
        "fmt"
        "bytes"
        "os/exec"
)

func ExecCommand(script string) ([]byte, error) {
        var buf bytes.Buffer
        cmd := exec.Command("sh", "-c", script)
        cmd.Stdout = &buf
        cmd.Stderr = &buf
        err := cmd.Start()
        if err != nil {
                return buf.Bytes(), err 
        }   
        _, err = cmd.Process.Wait()
        return buf.Bytes(), err 
}

func main() {
        start_num := 118261738305877
        for i:=0; i<50000; i++{
            buf, err := ExecCommand(fmt.Sprintf("echo 'Y = %d' | grep 'Y' | awk '{print $3}'", start_num))
            if err != nil {
                fmt.Println("cmd execute err at round", i)
            } else {
                if len(buf) == 0 { 
                    fmt.Println("cmd return nothing at round", i)
                } else {
                    if i % 100 == 0 { 
                        fmt.Println("cmd return ", string(buf), " at round ", i)
                    }   
                }   
            }   
        }   
}

Here is my output on linux:

# go run test_go_process.go
cmd return  118261738305877
  at round  0
cmd return  118261738305877
  at round  100
cmd return nothing at round 115
cmd return  118261738305877
  at round  200
cmd return  118261738305877
  at round  300
cmd return  118261738305877
  at round  400
cmd return  118261738305877
  at round  2800
cmd return  118261738305877
  at round  2900
cmd return  118261738305877
  at round  3000
cmd return  118261738305877
  at round  3100
cmd return  118261738305877
  at round  3200
cmd return nothing at round 3290
cmd return  118261738305877
  at round  3300
cmd return  118261738305877

output:cmd return nothing at round 115 shows that at this run, nothing return from func ExecCommand and I am wondering why and how to fix that except edit cmd.Process.wait(). Thanks a lot in advance.

user_1726083 · Answer

needs to run script "service mysql start" [ ... ] If I used cmd.Wait(), my application would wait for mysql to finish and that is not what I want, so I use cmd.process.wait.

That's not accurate, and it's the thing causing your behavior.

First of all, the service mysql start process does not run until mysql finishes, only until it starts. Otherwise it would never return when you ran it at the command line.

Second, os.Process.Wait and exec.Command.Wait both do wait for the process to complete. In fact, exec.Command.Wait calls os.Process.Wait.

The difference is that os.Process.Wait

Wait waits for the Process to exit, and then returns a ProcessState describing its status and an error, if any.

Whereas exec.Command.Wait (emphasis added)

Wait waits for the command to exit and waits for any copying to stdin or copying from stdout or stderr to complete.

In your program, you're omitting the synchronization required to make sure that your process's output is fully consumed. Without this synchronization, sometimes it may not be, and sometimes it may be.

output:cmd return nothing at round 115 shows that at this run, nothing return from func ExecCommand

If I run your code in the playground, the results are different, with missing stdout data in every case. (Probably because your environment has more available CPU concurrency than the go playground environment).

But if I change to cmd.Wait(), it works every time.

The Go playground isn't willing to let you invoke 50,000 subshell processes, but it's enough that you see the difference in behavior.

Go exec.Command do not return output to stdout when I use cmd.Process.wait()

1 Answers