How to share (or exclude from sharing) global variables over multiprocess?

user_3240688 0
Viewed 422

In Linux, I understand whenever you fork a child process, the child process gets a copy of the parent process address space.

I don't understand why this is always desirable. The parent process may have global objects that should not be shared between parent and child process. Why do we want child process to always get them?

For example, let's say parent process maintains a global database client.

client.py

client = None
def get_client():
   global client
   if client is None:
       client = make_new_db_client()
   return client

This works fine in my single process application. But could break once I start using multiprocessing.

The problem is that I'm sharing db_client object across processes. This database code is actually a 3rd party library. And let's pretend for a second it simply does not work across multiprocesses. So, I create a db_client in the parent process, and I'm forcing the child process to use it. It'd fail.

I think the right thing to do is to allow user to selectively pick what gets shared across multiprocess. So in this case, I simply won't share the db_client. Child process would start up and create its own database connection. Everything would work.

But that's not the way Python is designed. global variables are always copied (at least in Linux). Why is it desirable to do that? That seems like it's good sometimes but not always.

Or is there already a way to selectively hide variables from child process?

python-multiprocessing multiprocessing linux python
1 Answers

It isn't necessarily always desirable. In many cases it is irrelevant and in some cases confusing. Many parallel programming mistakes are related to diverging variables and the developer assuming they would be in sync.

Desirability has little to do with this design. In Linux (and most other Unix variants) Python interpreter creates a subprocess by calling system call fork() on C level. This creates an (almost) identical carbon copy of the process calling fork. There are some process bound resources, for example timers, that are not inherited by the child but variables are.

This is the traditional Unix way of creating a process and there are some benefits in it

  • The implementation (for those who have developed Python) is there provided by a system call that has survived the test of time.
  • It is fast. Very little logic is included in this - memory is just copied over to another location
  • Subprocess is viable. In a theoretical mix & match model described in your question it would be possible to create a subprocess that is missing a vital component
  • This offers a rapid way of passing data to a subprocess

Consider a subprocess that would not do this. It would be a tabula rasa. By default an empty core capable of receiving executable code and parameters somehow later from the parent that created it.

To make this a running and useful component would require much more work. Both in implementation (you would need to consider every detail) and also execution. Transferring data would require communication between processes. Initialising the process state to where you want it to be could theoretically require a lot of work, for example reading configuration parameters from a database. And if you need to launch thousands of subprocesses, repeating this work in every process would be much slower.

There are of course situations where inheriting global variables will lead you astray. Some developers shun global variables for this very reason. Connectors to external processes or services need to be recreated in children and if global variables represent these, your program can become messy.

One of the reasons parallel programming requires a good design and much attention to detail is exactly this - no programming language can guess what your program wants to do in a subprocess. What are the inputs and outputs? Does it need to communicate with the parent or another process or computer? How does this happen? Etc.

It is entirely possible parallel processing is not suitable to your task. There are many cases where subprocesses do not help you at all because of the functionality of your application. There are also occasions where apparent multiprocessing related problems can be fixed by redesigning your application. Sometimes it could be wise to handle external communications in the main process and only dispatch calculations etc. to subprocesses. In another case it could be ideal to have the main program do nothing or almost nothing, have every subprocess subscribe independently to external sources and deliver results directly. Many server side applications work this way.

Apologies for a lengthy answer that is almost a non-answer.

user_22778150
Related