Is `preexec_fn` ever safe in multi-threaded programs? Under what circumstances?

user_2146491 0
Viewed 154

I understand that using subprocess.Popen(..., preexec_fn=func) makes Popen thread-unsafe, and might deadlock the child process if used within multi-threaded programs:

Warning: The preexec_fn parameter is not safe to use in the presence of threads in your application. The child process could deadlock before exec is called. If you must use it, keep it trivial! Minimize the number of libraries you call into.

Are there any circumstances under which it is actually safe to use it within a multi-threaded environment? E.g. would passing a C-compiled extension function, one that does not acquire any interpreter locks by itself, be safe?

I looked through the relevant interpreter code and am unable to find any trivially occurring deadlocks. Could passing a simple, pure-Python function such as lambda: os.nice(20) ever make the child process deadlock?

Note: most of the obvious deadlocks are avoided via a call to PyOS_AfterFork_Child() (PyOS_AfterFork() in earlier versions of Python).

Note 2: for the sake of making the question answerable, lets assume we are running on a recent version of Glibc.

cpython subprocess python-3.x multithreading python
1 Answers

The following explanation is for POSIX only.

This issue of executing code after forking and before execing in a multi-threaded process is not Python specific.

In the child, do not call any library functions after calling fork() and before calling exec(). One of the library functions might use a lock that was held in the parent at the time of the fork().

Besides the usual concerns such as locking shared data, a library should be well behaved with respect to forking a child process when only the thread that called fork() is running. The problem is that the sole thread in the child process might try to grab a lock held by a thread not duplicated in the child.

For example, assume that T1 is in the middle of printing something and holds a lock for printf(), when T2 forks a new process. In the child process, if the sole thread (T2) calls printf(), T2 promptly deadlocks.

https://docs.oracle.com/cd/E19120-01/open.solaris/816-5137/gen-1/index.html

The fork( ) system call creates an exact duplicate of the address space from which it is called, resulting in two address spaces executing the same code.

Suppose that one of the other threads (any thread other than the one doing the fork( )) has the job of deducting money from your checking account.

POSIX defined the behavior of fork( ) in the presence of threads to propagate only the forking thread.

If the other thread has a mutex locked, the mutex will be locked in the child process, but the lock owner will not exist to unlock it. Therefore, the resource protected by the lock will be permanently unavailable.

The fact that there may be mutexes outstanding only becomes a problem if your code attempts to lock a mutex that could be locked by another thread at the time of the fork( ). This means that you cannot call outside of your own code between the call to fork( ) and the call to exec( ). Note that a call to malloc( ), for example, is a call outside of the currently executing application program and may have a mutex outstanding.

if your code calls some of your own code that does not make any calls outside of your code and does not lock any mutexes that could possibly be locked in another thread, then your code is safe.

http://www.doublersolutions.com/docs/dce/osfdocs/htmls/develop/appdev/Appde193.htm

When duplicating the parent process, the fork subroutine also duplicates all the synchronization variables, including their state. Thus, for example, mutexes may be held by threads that no longer exist in the child process and any associated resource may be inconsistent.

https://www.ibm.com/docs/en/aix/7.2?topic=programming-process-duplication-termination

https://pubs.opengroup.org/onlinepubs/000095399/functions/fork.html

https://lwn.net/Articles/674660/

https://softwareengineering.stackexchange.com/questions/384505/why-would-cpython-logging-use-a-lock-for-each-handler-rather-than-one-lock-per-l

The code below is stolen from https://blog.actorsfit.com/a?ID=00001-993928f7-96b0-42dd-8903-18ae712467f3

The preexec_fn in subprocess.Popenis similar to target in multiprocessing.Process and interrupting multiprocessing.Process clearly shows in the stacktrace the lock acquiring code (self.lock.acquire()). time.sleep in emit mostly results in deadlock.

import sys
import time
import logging
import threading
import multiprocessing
import subprocess

class MyHandler(logging.StreamHandler):
    def emit(self, record):
        time.sleep(0.1)
        super().emit(record)

logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
handler = MyHandler()
formatter = logging.Formatter('%(asctime)s %(process)d %(thread)d %(message)s')
handler.setFormatter(formatter)
logger.addHandler(handler)

def thread_fn():
    logger.info('thread')
    logger.info('thread')

def process_fn():
    logger.info('child')
    logger.info('child')

t1 = threading.Thread(target=thread_fn)
t1.start()

p1 = multiprocessing.Process(target=process_fn)
p1.start()

# subprocess.Popen(args=['echo from shell'], shell=True, preexec_fn=process_fn)

Normal output;

2022-06-30 03:28:28,162 30093 140582533375744 thread
2022-06-30 03:28:28,164 30100 140582559856448 child
2022-06-30 03:28:28,263 30093 140582533375744 thread
2022-06-30 03:28:28,266 30100 140582559856448 child

os.register_at_fork (pthread_atfork) was added in Python 3.7. This is the mechanism CPython uses to avoid deadlocks. https://github.com/google/python-atfork

For demo, I delete it before importing logging.

import os

del os.register_at_fork

# same code as above

Deadlock output;

2022-06-30 03:30:10,090 30374 140014242154240 thread
2022-06-30 03:30:10,191 30374 140014242154240 thread
^CError in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/popen_fork.py", line 27, in poll
Process Process-1:
    pid, sts = os.waitpid(self.pid, flag)
KeyboardInterrupt
Traceback (most recent call last):
  File "/usr/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap
    self.run()
  File "/usr/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/niz/src/python/tor-sec/tor_sec/new_fork_trhead.py", line 29, in process_fn
    logger.info('child')
  File "/usr/lib/python3.8/logging/__init__.py", line 1434, in info
    self._log(INFO, msg, args, **kwargs)
  File "/usr/lib/python3.8/logging/__init__.py", line 1577, in _log
    self.handle(record)
  File "/usr/lib/python3.8/logging/__init__.py", line 1587, in handle
    self.callHandlers(record)
  File "/usr/lib/python3.8/logging/__init__.py", line 1649, in callHandlers
    hdlr.handle(record)
  File "/usr/lib/python3.8/logging/__init__.py", line 948, in handle
    self.acquire()
  File "/usr/lib/python3.8/logging/__init__.py", line 899, in acquire
    self.lock.acquire()
KeyboardInterrupt

logging module at GH

user_45227800
Related