Python multiprocessing PicklingError on code taken directly from the documentation

Viewed 73

I am literally running the code directly from the python documentation https://docs.python.org/3/library/multiprocessing.html

from multiprocessing import Pool


def f(x):
    return x * x


if __name__ == '__main__':
    with Pool(5) as p:
        print(p.map(f, [1, 2, 3]))

And I get this error

Traceback (most recent call last):
  File "C:\Users\NAME\AppData\Local\Programs\Python\Python37\lib\site-packages\IPython\core\interactiveshell.py", line 3553, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-745c8e04055a>", line 1, in <module>
    runfile('C:/Users/NAME/PycharmProjects/RLS-Features-IQP/test.py', wdir='C:/Users/NAME/PycharmProjects/RLS-Features-IQP')
  File "C:\Program Files\JetBrains\PyCharm 2022.1\plugins\python\helpers\pydev\_pydev_bundle\pydev_umd.py", line 198, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "C:\Program Files\JetBrains\PyCharm 2022.1\plugins\python\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "C:/Users/NAME/PycharmProjects/RLS-Features-IQP/test.py", line 8, in <module>
    print(p.map(f, [1, 2, 3]))
  File "C:\Users\NAME\AppData\Local\Programs\Python\Python37\lib\multiprocessing\pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "C:\Users\NAME\AppData\Local\Programs\Python\Python37\lib\multiprocessing\pool.py", line 657, in get
    raise self._value
  File "C:\Users\NAME\AppData\Local\Programs\Python\Python37\lib\multiprocessing\pool.py", line 431, in _handle_tasks
    put(task)
  File "C:\Users\NAME\AppData\Local\Programs\Python\Python37\lib\multiprocessing\connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "C:\Users\NAME\AppData\Local\Programs\Python\Python37\lib\multiprocessing\reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <function f at 0x0000023F002A4678>: attribute lookup f on __main__ failed

I have tried everything from pathos multiprocessing, from pools to threads and I am out of ideas. What can I try next?

EDIT: https://stackoverflow.com/questions/70947312/multiprocessing-in-pycharm-results-in-pickle-picklingerror

1 Answers

From the attached output, I believe that the issue you are experiencing is due to pickle's difficulty to serialize interactive functions.

Indirectly, this limitation is actually mentioned on the documentation page you have provided:

Note: Functionality within this package requires that the main module be importable by the children. This is covered in Programming guidelines however it is worth pointing out here. This means that some examples, such as the multiprocessing.pool.Pool examples will not work in the interactive interpreter.

Saving this code in a Python file and running it using a standard interpreter should provide the results you are expecting.

Related