I'm trying to get shared_memory to work in python 3.8, and have run into the known bug where accessing a shared memory block with an independent process(not a child process) closes the shared memory block. I have seen turicas monkey patch, but am looking for a workaround in standard python. I believe there are other tools to achieve this since process 2 is read-only.
Minimum reproducible example: Process 1:
#process1.py
import numpy as np
from multiprocessing import shared_memory
import time
if __name__ == '__main__':
arr = np.random.rand(2,2,2)
shm = shared_memory.SharedMemory(name='foo',create=True, size=arr.nbytes)
buff = np.ndarray(arr.shape, dtype=arr.dtype, buffer=shm.buf)
while 1:
arr = np.random.rand(2, 2, 2)
buff[:] = arr[:]
time.sleep(1)
Process 2:
#process2.py
import numpy as np
from multiprocessing import shared_memory
import time
import sys
if __name__ =='__main__':
existing_shm = shared_memory.SharedMemory(name='foo')
for _ in range(3):
buff = np.ndarray((2,2,2), dtype=np.dtype('float64'), buffer=existing_shm.buf)
print(buff)
time.sleep(1.5)
existing_shm.close()
sys.exit()
Terminal window 1:
python process1.py
Terminal window 2:
python process2.py
[[[0.71427902 0.46214394]
[0.6427533 ...
## runs as expected, closing with warning:
_tracker.py:216: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
python process2.py
FileNotFoundError: [Errno 2] No such file or directory: '/foo'
Again, this is a known bug, but I am hoping for someone to point me in the direction to re-write this code (likely with something other than shared_memory).