File Descriptors closing in forked subprocesses in Python

user_854603 0
Viewed 118

In a Python 3.8 application a parent process is responsible for creating child processes (workers) and forwarding messages to them via IPC (tasks).

Parent process has its own logger that uses FileHandler to write to a file main.log but it also opens one logger per child process with a unique file for each of them to log IPC activity and errors.

Child processes are created via system call fork. IPC is done via OS queues. See below diagram that illustrates the situation.

childprocesses

The problem: while it is beneficial to log IPC activity on the parent process side and keep it in a dedicated file, this opens a lot of File Descriptors (FDs) which are all inherited by each and every subsequent child process.

Question: how to manage these FDs? Manually find and close them in each child process on run()? Is there a better way?

multiprocessing fork python
2 Answers

You can pass close_fds=True to whatever function (e.g. run) you are using for starting a subprocess:

If close_fds is true, all file descriptors except 0, 1 and 2 will be closed before the child process is executed. Otherwise when close_fds is false, file descriptors obey their inheritable flag as described in Inheritance of File Descriptors.

Source: https://docs.python.org/3/library/subprocess.html#subprocess.Popen

user_14125340

Set the close-on-exec flag for those file descriptors you don't want to carry over to the child processes. Either by setting the O_CLOEXEC flag upon opening, or later with fcntl set FD_CLOEXEC.

Before you ask, O_CLOEXEC != FD_CLOEXEC.

user_5243680
Related