How do I put an existing object into shared memory?

Question

I've managed to create an object that can exist in shared memory with BaseManager and NamespaceProxy, however all the examples I've seen require me to create the object with the proxy. For example:

class Foo:
    def __init__(self):
        self._a = 1000
    def get_a(self):
        return self._a

class SharedFoo(NamespaceProxy):
    _exposed_ = ('__getattribute__', '__getattr__', '__setattr__', '__init__', 'get_a')
    def get_a(self):
        callmethod = object.__getattribute__(self, '_callmethod')
        return callmethod('get_a', ())

class FooManager(BaseManager):
    pass

def test():
    FooManager.register('Foo', Foo, SharedFoo)
    with FooManager() as manager:
        ls = []
        t = time.time()
        for i in range(100):
            ls.append(manager.Foo())
        print(time.time() - t)

which prints out:

0.44 (and some other numbers that i ommitted)

Since I would likely create millions of Foo objects, this is too slow for the task. I tried to make it faster like this:

def do_stuff(obj):
    obj.ls[4].set_a(300)
def test():
    FooManager.register('Foo', Foo, SharedFoo)
    with FooManager() as manager:
        if manager._Server != None:
            manager._Server.mutex = NoLock()
        ls = []
        t = time.time()
        for i in range(100000):
            ls.append(Foo())
        foos = manager.Foo()
        foos.ls = mp.Manager().list(ls)
        print(time.time() - t)
    processes = [Process(target=do_stuff, args = (foos,)) for _ in range(3)]
    for process in processes:
        process.start()

    for process in processes:
        process.join()
    print(foos.ls[4].get_a())

which gave me this error:

Traceback (most recent call last):
  File "/path/to/lib/python3.7/multiprocessing/managers.py", line 788, in _callmethod
    conn = self._tls.connection
AttributeError: 'ForkAwareLocal' object has no attribute 'connection'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/path/to/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/path/to/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/path/to/myproject/test.py", line 121, in do_stuff
    obj.ls[4].set_a(300)
  File "/path/to/lib/python3.7/multiprocessing/managers.py", line 1099, in __getattr__
    return callmethod('__getattribute__', (key,))
  File "/path/to/lib/python3.7/multiprocessing/managers.py", line 792, in _callmethod
    self._connect()
  File "/path/to/lib/python3.7/multiprocessing/managers.py", line 779, in _connect
    conn = self._Client(self._token.address, authkey=self._authkey)
  File "/path/to/lib/python3.7/multiprocessing/connection.py", line 492, in Client
    c = SocketClient(address)
  File "/path/to/lib/python3.7/multiprocessing/connection.py", line 619, in SocketClient
    s.connect(address)
FileNotFoundError: [Errno 2] No such file or directory

Is what I'm trying to do possible? If so what should I use (not looking for a complete solution, just some ressources on how to do it)? I'm using Python 3.7 on Linux if that's relevant

Thanks

Edit: is this feasible with mmap, or am I going in a completely wrong direction? It looks promising but the documentations seems to say that it's more for files (again not looking for a complete solution, just if mmap would work with custom objects)

How do I put an existing object into shared memory?

0 Answers