Mysterious pickle error while profiling a multi-process Python script

Viewed 1960

I"m using the multiprocessing module, and I'm using UpdateMessage objects (my own class), sent through multiprocessing.Queue objects, to communicate between processes. Here's the class:

class UpdateMessage:
    def __init__(self, arrayref, rowslice, colslice, newval):
        self.arrayref = arrayref
        self.rowslice = rowslice
        self.colslice = colslice
        self.newval = newval
    def do_update(self):
        if self.arrayref == 'uL':
            arr = uL
        elif self.arrayref == 'uR':
            arr = uR
        else:
            raise Exception('UpdateMessage.arrayref neither uL nor uR')
        arr[self.rowslice, self.colslice] = self.newval

When I run the script, it works perfectly fine. However, when I run it with either cProfile or profile, it gives the following error:

_pickle.PicklingError: Can't pickle <class '__main__.UpdateMessage'>: attribute lookup __main__.UpdateMessage failed

It seems to be trying to pickle the class, but I can't see why this is happening. My code doesn't do this, and it works just fine without it, so it's probably the multiprocessing module. But why would it need to pickle UpdateMessage, and how can I fix the error?

EDIT: here's part of the code that's sending the UpdateMessage (multiple parts of the script do this, but all in the same way):

msg = UpdateMessage(uLref, refer[0] + marker[0] - 2,
                    slice(uL.shape[1]), ustar.copy())
queue.put(msg)

The traceback isn't very helpful:

Traceback (most recent call last):
  File "/usr/lib/python3.2/multiprocessing/queues.py", line 272, in _feed
    send(obj)
3 Answers
Related