Read-only Shared Memory across multiple Programs in Python

user_1175065 0
Viewed 301

I want to load data to memory once and want other processes to access (read-only) this data over the time. These processes are basically different python programs which are invoked at different time (ofcourse after loading the data).

In order to achieve this functionality, I am using shared memory. Please see following code snippet:

server.py

import numpy as np
from multiprocessing import shared_memory


class DataUploader:
    def __init__(self, shared_memory_name):
        # let's share the following two numpy arrays
        self._uint_np = np.random.randint(0, 255, size=(64, 4, 28, 28)).astype(np.uint8)
        self._float_np = np.random.rand(64, 8).astype(np.float32)

        name_1 = f"{shared_memory_name}_uint_np"
        name_2 = f"{shared_memory_name}_float_np"

        self._shm_1 = shared_memory.SharedMemory(name=name_1, create=True, size=self._uint_np.nbytes)
        self._shm_2 = shared_memory.SharedMemory(name=name_2, create=True, size=self._float_np.nbytes)

        # now create a numpy array backed by shared memory
        self._shared_1 = np.ndarray(self._uint_np.shape, dtype=self._uint_np.dtype, buffer=self._shm_1.buf)
        self._shared_2 = np.ndarray(self._float_np.shape, dtype=self._float_np.dtype, buffer=self._shm_2.buf)

        # copy the original data into shared memory
        self._shared_1[:] = self._uint_np[:]
        self._shared_2[:] = self._float_np[:]

    def __del__(self):
        if self._shm_1 is not None and self._shm_2 is not None:
            self._shm_1.close()
            self._shm_1.unlink()
            self._shm_2.close()
            self._shm_2.unlink()
            print("Shared memory destroyed")


if __name__ == "__main__":
    data_uploader = DataUploader(shared_memory_name="test")
    # keep running the program forever
    input(f'Press "enter" key to exit: ')

client.py

import numpy as np
from multiprocessing import shared_memory


class DataProvider:
    def __init__(self, shared_memory_name):
        self._existing_shm_1 = shared_memory.SharedMemory(name=f"{shared_memory_name}_uint_np")
        self._existing_shm_2 = shared_memory.SharedMemory(name=f"{shared_memory_name}_float_np")

        self._uint_np = np.ndarray((64, 4, 28, 28), dtype=np.uint8, buffer=self._existing_shm_1.buf)
        self._float_np = np.ndarray((64, 8), dtype=np.float32, buffer=self._existing_shm_2.buf)

    def get_item(self, idx):
        uint_np = self._uint_np[idx]
        float_np = self._float_np[idx]
        return uint_np, float_np

    def __del__(self):
        if self._existing_shm_1 is not None and self._existing_shm_2 is not None:
            self._existing_shm_1.close()
            self._existing_shm_2.close()


if __name__ == "__main__":
    data_provider = DataProvider(shared_memory_name="test")
    uint_np, float_np = data_provider.get_item(0)

    # just print some information about the accessed data
    print(uint_np.std(), float_np.std())

After executing, server.py once, I wish to execute client.py many times to access (read-only) the data. However, after first execution of client.py, following warning appears:

$ python client.py 
73.84145388455019 0.25972846
/home/ravi/tools/anaconda/envs/py39/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 2 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

From the second run, client.py throws following error:

$ python client.py 
Traceback (most recent call last):
  File "/home/ravi/test/client.py", line 34, in <module>
    data_provider = DataProvider(shared_memory_name="test")
  File "/home/ravi/test/client.py", line 16, in __init__
    self._existing_shm_1 = shared_memory.SharedMemory(name=f"{shared_memory_name}_uint_np")
  File "/home/ravi/tools/anaconda/envs/py39/lib/python3.9/multiprocessing/shared_memory.py", line 103, in __init__
    self._fd = _posixshmem.shm_open(
FileNotFoundError: [Errno 2] No such file or directory: '/test_uint_np'

Clearly the shared memory is destroyed / unreachable after first access.

OS Information:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.6 LTS
Release:    18.04
Codename:   bionic

$ uname -r
5.4.0-86-generic

Is there a way to keep alive the shared memory and access (read-only) it from different program multiple times?

shared-memory ubuntu python-3.x numpy
1 Answers

I found the answer in this thread. close() is thought for child processes, not for independent processes, therefore after close() the shared object is not unregistered from the resource_tracker. The solution is to close them manually:

from multiprocessing import resource_tracker
...

def __del__(self):
    if self._existing_shm_1 is not None and self._existing_shm_2 is not None:
        self._existing_shm_1.close()
        self._existing_shm_2.close()
        resource_tracker.unregister(self._existing_shm_1._name, "shared_memory")
        resource_tracker.unregister(self._existing_shm_2._name, "shared_memory")

Note: resource_tracker.unregister(self._existing_shm_2.name, "shared_memory") is not working, "\" is missing.

Edit: I would say there is no way to keep the shared_memory alive without registering manually from the resurce_tracker. At the moment, the resource_tracker will always remove the leaked shared memory.

user_33275430
Related