Unable to pickle redis-py instance (_thread.lock)

user_9451157 0
Viewed 552

I'm using Redis's client python implementation from (the de facto standard): https://pypi.org/project/redis/

So I'm defining multiple workers in the background, and each worker has a connection instance created on startup:

class Worker(Process):
    _db = None

    def __init__(self):
        super(Worker, self).__init__()
        self._db = redis.Redis(host="1.2.3.4", port=1234, db=0)

However, whenever I try to start an instance of this worker, I get the following error message:

TypeError: can't pickle _thread.lock objects

So I'm guessing that this implementation uses a lock somewhere. What is the workaround for this issue?

python-multiprocessing redis python-3.x multiprocessing python
1 Answers

You wouldn't have this issue on a Unix-y OS with forking, but it looks like a cheap workaround for Windows would be, to delay the redis instantiation until run is called in the new process:

from multiprocessing import Process
import redis

class Worker(Process):
    _db = None
    def __init__(self):
        super().__init__()
        self._db = None
    def run(self):
        self._db = redis.Redis(host='localhost', port=6379, db=0)
        # do stuff

if __name__ == '__main__':
    w = Worker()
    w.start()
    w.join()
user_90594200
Related