pool.map works on WIndows, not on Linux

Question

This is my code (it is a class whose job is to save to a MS Sharepoint folder some Excel files).


    class BackupOperation:
     
        def __init__(self, sharepoint: Sharepoint):
            self.sharepoint = sharepoint
    
        def run(self, names: list[str) -> None:
         
            pool = multiprocessing.Pool(processes=8)
            pool.map(self._backup_name, names)
            pool.close()
            pool.join()
        
        def _backup_name(self, name: str) -> None:
            # do things with Office365-REST-Python-Client
            # ...

My code runs smoothly on Windows 11, but now I have deployed it in a container that runs Linux and it outputs the following error: "AttributeError: Can't pickle local object 'ClientRuntimeContext.after_query_execute.._process_response'"

This is the full traceback:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/runpy.py", line 197, in _run_module_as_main
  return _run_code(code, main_globals, None,
  File "/usr/local/lib/python3.9/runpy.py", line 87, in _run_code
  exec(code, run_globals)
  File "/application/dcr-0.0.0/dcr/__main__.py", line 6, in 
  Trigger.run()
  File "/application/dcr-0.0.0/dcr/tools/trigger/trigger.py", line 47, in run
  Pipeline.run()
  File "/application/dcr-0.0.0/dcr/controller/pipeline.py", line 55, in run
  SharepointBackupper.run()
  File "/application/dcr- 
 0.0.0/dcr/controller/steps/sharepoint_backupper/sharepoint_backupper.py", line 31, in run
backup_operation.run(names=names)
  File "/application/dcr-0.0.0/dcr/controller/steps/sharepoint_backupper/backup_operation.py", line 37, in run
pool.map(self._backup_name, names)
  File "/usr/local/lib/python3.9/multiprocessing/pool.py", line 364, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/local/lib/python3.9/multiprocessing/pool.py", line 771, in get
raise self._value
  File "/usr/local/lib/python3.9/multiprocessing/pool.py", line 537, in _handle_tasks
put(task)
  File "/usr/local/lib/python3.9/multiprocessing/connection.py", line 211, in send
self._send_bytes(_ForkingPickler.dumps(obj))
  File "/usr/local/lib/python3.9/multiprocessing/reduction.py", line 51, in dumps
cls(buf, protocol).dump(obj)
  AttributeError: Can't pickle local object 'ClientRuntimeContext.after_query_execute.._process_response'

Why is this happening? From the traceback we can see that the problem arises in the moment it executes pool.map(self._backup_name, names)

pool.map works on WIndows, not on Linux

0 Answers