I want to share two large numpy array (about 6GB) between some processes. So I tried creating they array with sharedmem.empty() and then copy the data to it, but I faced an error and can't solve it:
Code:
# data is a numpy array
shared_data = sharedmem.empty(data.shape)
numpy.copyto(shared_data, data)
del data
This code works with simple data:
>>> a = numpy.array([1,2,3,4,5])
>>> b = sharedmem.empty(5)
>>> b
anonymousmemmap([0., 0., 0., 0., 0.])
>>> numpy.copyto(b, a)
>>> b
anonymousmemmap([1., 2., 3., 4., 5.])
But in real data, I faced this:
File "Python38\lib\site-packages\sharedmem\sharedmem.py", line 891, in __new__
self = numpy.ndarray.__new__(subtype, shape, dtype=descr, buffer=mm, order=order)
TypeError: buffer is too small for requested array
I'm pretty sure I have enough memory for allocating this array (I used numpy.zeros with the same size).
How can I solve this ?
P.S I'm using python3.8 on windows, and I can't change it for other matters.