Subprocess dies when thread dies

Question

I have a program that triggers Python timers to spawn subprocesses. These subprocesses should be terminated once my program is terminated or killed. In order to do that, I'm using the "prctl hack", that sets which signal a child should receive once its parent dies. The undesired behaviour I get is: even though my main process is running, the children get killed. The following code recreates the problem:

from threading import Timer
import time
import os
import subprocess
import ctypes
import signal

def set_pdeathsig():
        print("child PID: %d" % os.getpid())
        print("child's parent PID: %d" % os.getppid())
        prctl = ctypes.CDLL("libc.so.6").prctl
        PR_SET_PDEATHSIG = 1
        prctl(PR_SET_PDEATHSIG, signal.SIGTERM)

def thread1():
        subprocess.Popen(['sleep', 'infinity'], preexec_fn=set_pdeathsig)
        time.sleep(10)
        print("thread 1 finished")

def thread2():
        subprocess.Popen(['sleep', 'infinity'], preexec_fn=set_pdeathsig)
        time.sleep(10)
        print("thread 2 finished")

print("main thread PID: %d" % os.getpid())

t1 = Timer(1, thread1)
t2 = Timer(1, thread2)

t1.start()
t2.start()

time.sleep(100)

You can notice that, before the threads die, the sleep processes are still running. After the timer threads die, its respective subprocess dies as well, even with the main thread alive.

user_2277815 · Answer

This is the expected and even documented behaviour. From the manual page of prctl(2):

      Warning: the "parent" in this case is considered to be the
      thread that created this process.  In other words, the signal
      will be sent when that thread terminates (via, for example,
      pthread_exit(3)), rather than after all of the threads in the
      parent process terminate.

This means you need to spawn your subprocesses elsewhere. If you do it in a thread that exits, then your subprocess dies as expected and there is no way around that.

I would add another thread and do process launching from there. Would something like this work:

from threading import Timer
from threading import Thread
import queue
import time
import os
import subprocess
import ctypes
import signal

def set_pdeathsig():
    print("child PID: %d" % os.getpid())
    print("child's parent PID: %d" % os.getppid())
    prctl = ctypes.CDLL("libc.so.6").prctl
    PR_SET_PDEATHSIG = 1
    prctl(PR_SET_PDEATHSIG, signal.SIGTERM)

def thread1(q):
    q.put(["sleep", "infinity"])
    time.sleep(5)
    print("thread 1 finished")

def thread2(q):
    q.put(["sleep", "infinity"])
    time.sleep(5)
    print("thread 2 finished")

def process_manager(q):
    while True:
        foo = q.get()
        subprocess.Popen(foo, preexec_fn=set_pdeathsig)

print("main thread PID: %d" % os.getpid())

qu = queue.Queue()
pm_thread = Thread(group=None, target=process_manager, args=(qu,))
pm_thread.daemon = True
pm_thread.start()


t1 = Timer(1, thread1, args=(qu,))
t2 = Timer(1, thread2, args=(qu,))

t1.start()
t2.start()

time.sleep(15)

This does what you want it to do (Python3.5 used to test). Of course there might be reasons why an orchestration thread would not be suitable, but I offer it as a solution candidate anyway. Now your subprocesses survive the death of Timer threads, but will still be terminated when your main thread exits.

Subprocess dies when thread dies

1 Answers