AttributeError: Can't pickle local object in Multiprocessing

user_19422691 0
Viewed 2682

I am very new to python and I encounter this error. CODE 1 :

import multiprocessing as mp
import os
 
def calc(num1, num2):
    global addi
    def addi(num1, num2):
        print(num1+num2)
    m = mp.Process(target = addi, args = (num1, num2))
    m.start()

    print("here is main", os.getpid())
    m.join()
  
if __name__ == "__main__":
    # creating processes
   calc(5, 6)

ERROR 1 :    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'calc.<locals>.addi'

After reading around a little I understand that pickle cannot be used for local methods and so I also tried the below solution which gave another error.

CODE 2 :

import multiprocessing as mp
import os
   
def calc(num1, num2):
    **global addi**
    def addi(num1, num2):
        print(num1+num2)
    m = mp.Process(target = addi, args = (num1, num2))
    m.start()

    print("here is main", os.getpid())
    m.join()
  
if __name__ == "__main__":
    # creating processes
   calc(5, 6)
ERROR 2 :
 self = reduction.pickle.load(from_parent)
AttributeError: Can't get attribute 'addi' on <module '__mp_main__' from '/Users

Could someone please help me out with this? I am clueless on what to do next! The python version I am using is python3.8.9

Thank you so much!

python-3.8 attributeerror multiprocessing
2 Answers

Basically, the reason you are getting this error is because multiprocessing uses pickle, which can only serialize top-module level functions in general. Function addi is not a top-module level function. In fact, the line global addi is not doing anything because addi has never been declared in the outer module. So you have two ways to fix this.

Method 1

You can define addi in the global scope before executing calc function:

import multiprocessing as mp
import os


def addi(num1, num2):
    print(num1 + num2)

def calc(num1, num2):

    m = mp.Process(target=addi, args=(num1, num2))
    m.start()

    print("here is main", os.getpid())
    m.join()


if __name__ == "__main__":
    # creating processes
    calc(5, 6)

Output

here is main 9924
11

Method 2

You can switch to multiprocess, which uses dill instead of pickle, and can serialize such functions.

import multiprocess as mp  # Note that we are importing "multiprocess", no "ing"!
import os

def calc(num1, num2):

    def addi(num1, num2):
        print(num1 + num2)

    m = mp.Process(target=addi, args=(num1, num2))
    m.start()

    print("here is main", os.getpid())
    m.join()


if __name__ == "__main__":
    # creating processes
    calc(5, 6)

Output

here is main 67632
11
user_163107410

set_start_method('fork') in main

user_193515930
Related