Understanding .fork() with multiprocessing and logging

user_12203802 0
Viewed 52

I am running a multiprocessing pool which logs to a logger that is configured by the main parent process. In one scenario, I have

def init():
  global LOG
  LOG = logging.getLogger(__name__)
  LOG.setLevel(logging.DEBUG)

def main():
  LOG.info("test")

if __name__ == '__main__':
  fmt = "%(asctime)s %(name)s %(levelname)s: %(message)s"
  logging.basicConfig(format=fmt, datefmt='%m/%d/%Y %I:%M:%S %p')
  with multiprocessing.Pool(initializer = init) as pool:
    pool.map(main, inputs)

Nothing gets printed to the stdout of my main calling thread. However, if I do this:

fmt = "%(asctime)s %(name)s %(levelname)s: %(message)s"
logging.basicConfig(format=fmt, datefmt='%m/%d/%Y %I:%M:%S %p')

def init():
  global LOG
  LOG = logging.getLogger(__name__)
  LOG.setLevel(logging.DEBUG)

def main():
  LOG.info("test")

if __name__ == '__main__':
  with multiprocessing.Pool(initializer = init) as pool:
    pool.map(main, inputs)

Then I do have proper logging. I don't see why these two things are different. When python forks into worker processes, the child process should be identical to the parent process so there shouldn't be a distinction between whether we called logging.basicConfig in the global namespace vs the main execution block. Can someone clarify?

logging multiprocessing python
1 Answers

Your code is perfectly fine for Linux-es (except the fact that it won't even run, haha)

However for Windows and OSX multiprocessing would spawn a fresh interpeter (see the docs) and would try to import the target module. This is the fork (pun intended) point where your __name__ == '__main__' guard makes a difference -- your logging setup is not done at all in the spawned workers!

user_5395700
Related