Unable to open `proc/[pid]/status` for live memory profiling of an executing command

Viewed 149

Motive:

I have been trying to profile code execution for time as well as memory consumption and kill the executing command once a limit for either parameter is reached.

Roadblock:

I am trying to write a go-routine that constantly monitors the process memory consumption by reading the status from proc/[pid]/status at every 100-millisecond interval.

Command Execution Code Block

// code block:

// define a resource group with limited memory
    rlimit := syscall.Rlimit{
        Cur: uint64(2048 * 1024 * 1024), // 1GB
        Max: uint64(2048 * 1024 * 1024), // 1GB
    }

// Create a new command
    cmd = exec.Command("/bin/bash", "-c", execCmd)

// Set command as separate process
    cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
    // Set resource limit
    err = syscall.Setrlimit(syscall.RLIMIT_AS, &rlimit)
    if err != nil {
        log.Fatal("Error in setting rlimit: ", err)
    }
    start := time.Now()

Memory Usage calculation Utility

Go Library I am using pidusage

// memoryUsage returns the memory usage of a process in bytes.
func memoryUsage(pid int) (float64, error) {
    stats, err := pidusage.GetStat(pid)
    if err != nil {
        return 0, err
    }
    return stats.Memory / 1024, nil
}

Go-routine

// call the function monitorMemory every 10 milli-seconds
// Get the current memory usage in the background
    go func(start time.Time) {
        // Print the memory usage every 10 milli-seconds
        for {
            // Get the current memory usage
            memoryUsage, err := memoryUsage(cmd.Process.Pid)
            if err != nil {
                fmt.Println(err)
                os.Exit(1)
            }

            // kill process if memory usage is greater than 2MB
            if memoryUsage > 500*1024 {
                err := syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL)
                if err != nil {
                    log.Fatal(err)
                    os.Exit(1)
                }
                result.ErrorCode = "MLE"
                result.RunTimeErr = "Memory Limit Exceeded. Took more than 500MB."
                result.MemoryUsage = memoryUsage
                result.RunTime = time.Since(start)
                printformattedResult(*result)

                // Exit the loop
                break
            }

            time.Sleep(1 * time.Nanosecond)
        }
    }(start)

The approach works fine If set a memory limit for less than 50 MB but once I increase the limit to monitor and wait for I start getting the error as:

open /proc/110050/stat: no such file or directory

Other Tried approaches:

I also tried reading memory usage as float64(cmd.ProcessState.SysUsage((*syscall.Rusage).Maxrss) but for some reason go routine throws an error as `memory address not found. I do not have very in-depth knowledge of go-routine but I assumed this might be due to it trying to access a child process that is running separately.

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x4bb794]

goroutine 33 [running]:
os.(*ProcessState).sysUsage(...)
        /usr/local/go/src/os/exec_posix.go:96
os.(*ProcessState).SysUsage(...)
        /usr/local/go/src/os/exec.go:175
main.main.func2({0x0, 0x0, 0x5a0360})
        /path/to/my/go/file.go:149 +0x74
created by main.main
        /path/to/my/go/file.go:145 +0x7cf
0 Answers
Related