The Plasma in-memory object store can store the NumPy array and some other types using shared memory so that when the same process or other processes need to use the array, Plasma can do so without any memory copy. The code example below verifies this.
from psutil import Process
import os
import numpy as np
from pyarrow import plasma
p = Process(os.getpid())
store = plasma.start_plasma_store(1024 * 1024 * 1024)
client = plasma.connect(store.__enter__()[0])
array = np.array([i for i in range(100000000)])
id = client.put(array)
# Take readings after putting the object in the store
memory_info_before = p.memory_info().rss
# Make 100 copies of the object
objects = []
for i in range(100):
objects.append(client.get(id))
# Take readings after making 100 copies of the object
memory_info_after = p.memory_info().rss
# Verify that the objects are the same
assert memory_info_before == memory_info_after
print(f"Used Memory {memory_info_after / 1e9} GB")
However, for the serialized objects, each copy of the unserialized version makes an actual copy of the object. We can verify this by trying to serialize the array using the pickle (we don't need to serialize the array, this is to show the issue).
serialized_array = pickle.dumps(array, pickle.HIGHEST_PROTOCOL)
id = client.put(serialized_array)
And make a copy using.
objects.append(pickle.loads(client.get(id)))
Both pickle and plasma allow zero-copy usage. So, is there any way to store the serialized version of the object into the plasma in-memory store using shared memory and doing zero-copy for each copy of the object? My application is not suitable for using ray. Thanks!