Goal

I wish to send a large dict of the following type to all python multiprocesses.

Type

thesaurus: Dict[str, Tuple[str, ...]] = {}

Obviously, the value of thesaurus is set elsewhere, but is completely determined before multiprocessing.

• None of the forked processes mutate the thesaurus, they only refer to it for their calculations.
• Importantly, thesaurus is very large, so sending it to child processes as a variable again and again makes no sense. The actual iterable variable sent to multiprocessing is pretty small.

NB

• The code is tested successfully by passing thesaurus (reduced size) as a variable.
• I'm using multiprocessing.Pool().map_async function for calculations.

Gist of tested code

#!/usr/bin/env python3
# -*- coding: utf-8; mode: python3 -*-
import os
import multiprocessing as mp
from typing import Dict, Tuple

thesaurus: Dict[str, Tuple[str, ...]] = {}
my_iter: Dict[str, str] = {}

# populate thesaurus
thesaurus['s_1'] = ('a', 's', 'd', 'f')
thesaurus['s_2'] = ('q', 'w', 'e')
thesaurus['s_3'] = ('a', 'q')
my_iter['id_1'] = 'asdf'
my_iter['id_2'] = 'qwerty'


def _worker(args) -> Tuple[str, Tuple[str, ...]]:
    m_thesaurus = args[0]
    some_id: str = args[1]
    var: str = args[2]
    worker_res = tuple((key
                        for key, val in m_thesaurus.items()
                        if any(s in var for s in val)))
    return some_id, worker_res


with mp.Pool(len(os.sched_getaffinity(0)) - 1 or 1) as pool:
    res_map = pool.map_async(_worker,
                             ((thesaurus, s_id, var)
                              for s_id, var in my_iter.items()))
    mapped_dicts = dict(res_map.get())

print(mapped_dicts)

Results

{'id_1': ('s_1', 's_3'), 'id_2': ('s_2', 's_3')}

Attempt

Since multiprocessing.SharedMemoryManager() supports ShareableList but no dict counterpart, I tried creating two ShareableList objects from thesaurus.keys() and thesaurus.values(), planning to iterate over them zipped inside the worker. ShareableList of thesaurus.keys() got created successfully.

Failure

However, ShareableList of thesaurus.values() can't be created, since it is a nested Iterable.

Attempted code

#!/usr/bin/env python3
# -*- coding: utf-8; mode: python3 -*-
import os
import multiprocessing as mp
from multiprocessing.managers import SharedMemoryManager
from typing import Dict, Tuple

thesaurus: Dict[str, Tuple[str, ...]] = {}
my_iter: Dict[str, str] = {}

# populate thesaurus
thesaurus['s_1'] = ('a', 's', 'd', 'f')
thesaurus['s_2'] = ('q', 'w', 'e')
thesaurus['s_3'] = ('a', 'q')
my_iter['id_1'] = 'asdf'
my_iter['id_2'] = 'qwerty'


def _worker(args) -> Tuple[str, Tuple[str, ...]]:
    m_keys: Tuple[str, ...] = args[0]
    m_vals: Tuple[Tuple[str, ...], ...] = args[1]
    some_id: str = args[2]
    var: str = args[3]
    worker_res = tuple((key
                        for key, val in zip(m_keys, m_vals)
                        if any(s in var for s in val)))
    return some_id, worker_res


with SharedMemoryManager() as smm:
    t_keys = smm.ShareableList(tuple(thesaurus.keys()))
    t_values = smm.ShareableList(tuple(thesaurus.values()))
    with mp.Pool(len(os.sched_getaffinity(0)) - 1 or 1) as pool:
        res_map = pool.map_async(_worker,
                                 ((t_keys, t_values, s_id, var)
                                  for s_id, var in my_iter.items()))
        mapped_dicts = dict(res_map.get())

print(mapped_dicts)

Traceback

Traceback (most recent call last):
  File "/home/pradyumna/./test_shared_multi_dict.py", line 32, in <module>
    t_values = smm.ShareableList(tuple(thesaurus.values()))
  File "/usr/lib64/python3.10/multiprocessing/managers.py", line 1372, in ShareableList
    sl = shared_memory.ShareableList(sequence)
  File "/usr/lib64/python3.10/multiprocessing/shared_memory.py", line 298, in __init__
    _formats = [
  File "/usr/lib64/python3.10/multiprocessing/shared_memory.py", line 299, in <listcomp>
    self._types_mapping[type(item)]
KeyError: <class 'tuple'>