Cause of `fork: retry: No child processes`

user_5819052 0
Viewed 17904

When I run a certain commercial build flow in OpenSuse, I encounter the following error:

fork: retry: No child processes

I encounter this error at various locations in the flow that corresponds with different executables, so I believe that the tool itself is not the problem. However, I run several instances (up to 16) of this build flow at the same time. The build is dispatched to different servers according to load, so sometimes a few builds run on the same machine. I suspect that the server load has somehow to do with this, but I cannot figure out exactly how.

I suspect that the error message corresponds with the EAGAIN error of fork. The fork manual gives several potential causes for this error:

  1. the RLIMIT_NPROC soft resource limit (set via setrlimit(2)), which limits the number of processes and threads for a real user ID, was reached
  2. the kernel's system-wide limit on the number of processes and threads, /proc/sys/kernel/threads-max, was reached (see proc(5))
  3. the maximum number of PIDs, /proc/sys/kernel/pid_max, was reached

I believe none of these are satisfied, because

  1. ulimit -a reports unlimited as the maximum number of user processes.
  2. /proc/sys/kernel/threads-max is set to 1031129. I monitor the number of lines in ps auxH, which should be the number of threads (plus the header line that ps outputs I guess), at a 1-second granularity, but it never gets higher than 917.
  3. /proc/sys/kernel/pid_max is set to 32768. I monitor the number of processes using ps aux, but it never gets higher than 726.

I believe I read somewhere that there is also a possibility that the kernel ran out of other (memory?) resources. How can I check that? Or is something else wrong in my line of reasoning?

The output form ulimit -a is:

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 515564
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 5120
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
linux-kernel multithreading fork linux c
1 Answers

Maybe it is obvious, but is the ulimit -a output for user that is running this build flow? Example output from one of my servers:

[root@server]# cat /etc/security/limits.d/20-nproc.conf
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.
*          soft    nproc     4096
root       soft    nproc     unlimited

Usually unlimited nr of processes is left only for root in order not to run in a fork bomb. Here all other users except root will have a limit.

Also, I see a limit on nr of open files. For a user you can check this with lsof -u <user>| wc -l

user_107227180
Related