How to set logging configuration when using python multiprocessing

Viewed 23

I'm trying to figure out why, when I attempt to log from a process forked using multiprocessing.Process, the log message doesn't honor the logging configuration set in the constructor, however, when I fork the same process using os.fork(), it does. Following is a pared-down code sample to demonstrate:

from multiprocessing import Process
import logging
import logging.config
import os
import sys

class Test():
    def __init__(self):
        formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
        ch = logging.StreamHandler()
        ch.setLevel(logging.DEBUG)
        ch.setFormatter(formatter)
        self.log = logging.getLogger(__name__)
        self.log.setLevel(logging.DEBUG)
        self.log.addHandler(ch)

    def test_log(self):
        self.log.warning(f'{os.getpid()}')

    def mp_test(self):
        self.log.warning('starting multiprocessing test')
        proc = Process(target=self.test_log)
        proc.start()
        proc.join()

    def fork_test(self):
        self.log.warning('starting fork test')
        if os.fork() == 0:
            self.test_log()
        else:
            os.wait()

if __name__ == '__main__':
    test = Test()
    test.mp_test()
    test.fork_test()

Output:

❯ python3 concurrent.py
2022-09-06 19:52:20,933 - __main__ - WARNING - starting multiprocessing test
68926
2022-09-06 19:52:20,977 - __main__ - WARNING - starting fork test
2022-09-06 19:52:20,978 - __main__ - WARNING - 68927

Why is the configuration for self.log not set anymore when self.test_log() is called by multiprocessing.Process? I was under the impression that multiprocessing.Process and os.fork() basically did the same thing and that the new process gets a copy of the parent's address space. I've probably misunderstood something but I'm really keen to find out what that is. Thanks.

0 Answers
Related