We've noticed a bunch of defunct (zombie) processes being leftover in one of our deployments and managed to produce a very minimal program showing the issue:
multi.py:
from multiprocessing import Pool, set_start_method
def f(x):
return x*x
if __name__ == '__main__':
set_start_method('spawn')
with Pool(5) as p:
print(p.map(f, [1, 2, 3]))
p.close()
p.join()
This program seems to be leaving zombie processes but it was hard to catch since running this from a regular shell will cause the shell to reap the zombies.
In our deployment we have this run from another python program, so to simulate it we have this:
main.py:
from subprocess import run
from time import sleep
while True:
result = run(["python", "multi.py"], capture_output=True)
print(result.stdout.decode('utf-8'))
result = run(["ps", "-ef", "--forest"], capture_output=True)
print(result.stdout.decode('utf-8'), flush=True)
sleep(1)
Running main.py produces this output:
[1, 4, 9]
UID PID PPID C STIME TTY TIME CMD
root 1 0 11 11:33 pts/0 00:00:00 python main.py
root 8 1 0 11:33 pts/0 00:00:00 [python] <defunct>
root 17 1 0 11:33 pts/0 00:00:00 ps -ef --forest
[1, 4, 9]
UID PID PPID C STIME TTY TIME CMD
root 1 0 6 11:33 pts/0 00:00:00 python main.py
root 8 1 3 11:33 pts/0 00:00:00 [python] <defunct>
root 19 1 0 11:33 pts/0 00:00:00 [python] <defunct>
root 28 1 0 11:33 pts/0 00:00:00 ps -ef --forest
[1, 4, 9]
UID PID PPID C STIME TTY TIME CMD
root 1 0 4 11:33 pts/0 00:00:00 python main.py
root 8 1 1 11:33 pts/0 00:00:00 [python] <defunct>
root 19 1 3 11:33 pts/0 00:00:00 [python] <defunct>
root 30 1 0 11:33 pts/0 00:00:00 [python] <defunct>
root 39 1 0 11:33 pts/0 00:00:00 ps -ef --forest
[1, 4, 9]
UID PID PPID C STIME TTY TIME CMD
root 1 0 3 11:33 pts/0 00:00:00 python main.py
root 8 1 1 11:33 pts/0 00:00:00 [python] <defunct>
root 19 1 1 11:33 pts/0 00:00:00 [python] <defunct>
root 30 1 4 11:33 pts/0 00:00:00 [python] <defunct>
root 41 1 0 11:33 pts/0 00:00:00 [python] <defunct>
root 50 1 0 11:33 pts/0 00:00:00 ps -ef --forest
On the other hand, the following program does not produce defunct processes:
main-sig.py:
from os import wait
import signal
from subprocess import run
from time import sleep
def chld_handler(_signum, _frame):
wait()
signal.signal(signal.SIGCHLD, chld_handler)
while True:
result = run(["python", "multi.py"], capture_output=True)
print(result.stdout.decode('utf-8'))
result = run(["ps", "-ef", "--forest"], capture_output=True)
print(result.stdout.decode('utf-8'), flush=True)
sleep(1)
Also the following simple shell script deoes not produce zombies:
#!/usr/bin/env bash
while :; do
python multi.py
ps -ef --forest
sleep 1
done
Is this a bug in Python or you are expected to handle any zombies coming from a child processes (like Bash is seemingly doing)?
All code and a Dockerfile to easily reproduce the issue are available here:
https://github.com/viktorvia/python-multi-issue
The issue is reproducible with Python 3.9.6, 3.7.4 and 3.7.11.