Python - How to share large dataset among multiple processes?

user_3827381 0
Viewed 253

I need to read a large dataset (about 25GB of images) into memory and read it from multiple processes. None of the processes has to write, only read. All the processes are started using Python's multiprocessing module, so they have the same parent process. They train different models on the data and run independently of each other. The reason why I want to read it only one time rather than in each process is that the memory on the machine is limited.

I have tried using Redis, but unfortunately it is extremely slow when many processes read from it. Is there another option to do this?

Is it maybe somehow possible to have another process that only serves as a "get the image with ID x" function? What Python module would be suited for this? Otherwise, I was thinking about implementign a small webserver using werkzeug or Flask, but I am not sure if that would become my new bottleneck then...

Another possibility that came to my mind was to use threads instead of processes, but since Python is not really doing "real" multithreading, this would probably become my new bottleneck.

multiprocessing python
2 Answers

If you are on linux and the content is read-only, you can use the linux fork inheriting mechanism.

from mp documentation:

Better to inherit than pickle/unpickle

When using the spawn or forkserver start methods many types from multiprocessing need to be picklable so that child processes can use them. However, one should generally avoid sending shared objects to other processes using pipes or queues. Instead you should arrange the program so that a process which needs access to a shared resource created elsewhere can inherit it from an ancestor process.

which means:

Before you fork your child processes, prepare your big data in a module level variable (global to all the functions).

Then in the same module, run your child with multiprocessing in 'fork' mode set_start_method('fork').

using this the sub-processes will see this variable without copying it. This happens due to linux forking mechanism that creates child processes with the same memory mapping as the parent (see "copy on write").

user_37006260

I'd suggest mmapping the files, that way they can be shared across multiple processes as well as getting swapped in/out as appropriate

the details of this would depend on what you mean by "25GB of images" and how these models want to access the images

the basic idea would be to preprocess the images into an appropriate format (e.g. one big 4D uint8 numpy array or maybe smaller ones, indicies could be (image, row, column, channel)) and save them in a format where they can be efficiently used by the models. see numpy.memmap for some examples of this

I'd suggest preprocessing files into a useful format "offline", i.e. not part of the model training but a seperate program that is run first. as this would probably take a while and you'd probably not want to do it every time

user_13583080
Related