Python - Multiprocessing Pool map returning can't pickle error

Viewed 1160

I have following code which creates a testrail client and executes testrail's GET_SUITES API call.

I have a function to call the GET_SUITES API and I am passing testrail client & test_rail_project_id as params

I am trying to use multiprocessing to execute over my list of projects to speed up things and I am can't pickle error

My code:

from itertools import product

def get_suites(client, project_id):
    try:
        path = 'get_suites/{projectid}'.format(projectid=project_id)
        test_rail_response = client.send_get(path)
        return test_rail_response
    except Exception as e:
        raise Exception(str(e))

if __name__ == "__main__":
    testRailClient = APIClient(TESTRAIL_URL)
    pool = Pool(2)
    all_project_ids = [100, 200, 300]
    data = pool.starmap(get_suites, product([testRailClient], all_project_ids))

Error stack:

Traceback (most recent call last):
  File "main.py", line 57, in <module>
    data = pool.starmap(testrailapi.get_suites, product([testRailClient], all_project_ids))
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/pool.py", line 274, in starmap
    return self._map_async(func, iterable, starmapstar, chunksize).get()
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/pool.py", line 644, in get
    raise self._value
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/pool.py", line 424, in _handle_tasks
    put(task)
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/usr/local/Cellar/python/3.6.5/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
TypeError: can't pickle SSLContext objects

Any suggestions please?

Thank you

PS: I am using Python3.6

UPDATE: As suggested I tried removing the API client as a parameter and it worked but I am getting the same error when I have "get_suites" as a method. Please see my updated code below

class TestRailExecution:

    def __init__(self, url, username, password):
        self.url = url
        self.username = username
        self.password = password
        self.client = APIClient(self.url)
        self.client.user = username
        self.client.password = password

    def get_suites(self, project_id):
        try:
            path = 'get_suites/{projectid}'.format(projectid=project_id)
            test_rail_response = self.client.send_get(path)
            return test_rail_response
        except Exception as e:
            raise Exception(str(e))

if __name__ == "__main__":
    testRailClient = TestRailExecution(TESTRAIL_URL, user, password)
    pool = Pool(2)
    data = pool.map(get_suites, [100, 200, 300])
0 Answers
Related