I don't think I fully understand the relationship between object_store_memory and redis_max_memory. I initially thought that the redis db holds all the objects in memory, but then how can the object store size be made larger than the redis_max_memory size? Or are some parts of it written to disk? How is RAM consumed? Is RAM_CONSUMED = object_store_memory + redis_max_memory, or is it the union of it?
1 Answers
TL;DR:
Yes, RAM_CONSUMED = object_store_memory + redis_max_memory, unless you back the object store by disk (allow for object spilling to disk).
More info
The redis DB only holds the metadata about objects and tasks (i.e. for tasks: which objects the task depends on and the IDs of the output it produces, which function needs to be run to produce the output; for objects: on which node(s) in the cluster is the object stored). The redis_max_memory restricts the size of this database, old entries are discarded in LRU fashion.
The actual data is stored in a shared memory object store (see https://arrow.apache.org/docs/python/plasma.html), the size of which is limited by object_store_memory. Again old objects that are not currently mapped into any worker memory are going to be evicted in LRU order from there.
It is also possible to back the object store with disk by providing the _plasma_directory argument or --plasma-directory switch (see docs of ray.init()). This allows to have object stores that are larger than the size of your RAM, but it will also make the object store slower, depending on the disk type and size and the amount of buffer cache.
Let me know if you have more questions.
Related