Python Implementation of the Object Pool Design Pattern

Oddthinking_8014 0
Viewed 12111

I need an Object Pool, and rather than implement it myself, I thought I would look around for a ready-made and tested Python library.

What I found was plenty of other people looking, but not getting many straight answers, so I have brought it over here to Stack Overflow.

In my case, I have a large number of threads (using the threading module), which need to occasionally call a remote SOAP-based server. They could each establish their own connection to the server, but setting up a socket and completing the authentication process is expensive (it is throttled by the server), so I want to share a pool of connections, creating more only as needed.

If the items to pool were worker subprocesses, I might have chosen multiprocessing.pool, but they are not. If they were worker threads, I might have chosen this implementation, but they are not.

If they were MySQL connections, I might have chosen pysqlpool, but they are not. Similarly the SQLAlchemy Pool is out.

If there was one thread, using a variable number of connections/objects, I would consider this implementation, but I need it to be thread-safe.

I know I could implement this again fairly quickly, but given there are many people looking for it, I thought a canonical answer on Stack Overflow would be nice.

data-structures design-patterns python
3 Answers

I had a similar problem and I must say Queue.Queue is quite good, however there is a missing piece of the puzzle. The following class helps deal with ensuring the object taken gets returned to the pool. Example is included.

I've allowed 2 ways to use this class, with keyword or encapsulating object with destructor. The with keyword is preferred but if you can't / don't want to use it for some reason (most common is the need for multiple objects from multiple queues) at least you have an option. Standard disclaimers about destructor not being called apply if you choose to use that method.

Hopes this helps someone with the same problem as the OP and myself.

class qObj():
  _q = None
  o = None

  def __init__(self, dQ, autoGet = False):
      self._q = dQ

      if autoGet == True:
          self.o = self._q.get()

  def __enter__(self):
      if self.o == None:
          self.o = self._q.get()
          return self.o
      else:
          return self.o 

  def __exit__(self, type, value, traceback):
      if self.o != None:
          self._q.put(self.o)
          self.o = None

  def __del__(self):
      if self.o != None:
          self._q.put(self.o)
          self.o = None


if __name__ == "__main__":
  import Queue

  def testObj(Q):
      someObj = qObj(Q, True)

      print 'Inside func: {0}'.format(someObj.o)

  aQ = Queue.Queue()

  aQ.put("yam")

  with qObj(aQ) as obj:
      print "Inside with: {0}".format(obj)

  print 'Outside with: {0}'.format(aQ.get())

  aQ.put("sam")

  testObj(aQ)

  print 'Outside func: {0}'.format(aQ.get())

  '''
  Expected Output:
  Inside with: yam
  Outside with: yam
  Inside func: sam
  Outside func: sam
  '''
user_6199330

For simple use cases here is an example implementation of the object pool pattern based on a list:

Source:

https://sourcemaking.com/design_patterns/object_pool

https://sourcemaking.com/design_patterns/object_pool/python/1

"""
Offer a significant performance boost; it is most effective in
situations where the cost of initializing a class instance is high, the
rate of instantiation of a class is high, and the number of
instantiations in use at any one time is low.
"""


class ReusablePool:
    """
    Manage Reusable objects for use by Client objects.
    """

    def __init__(self, size):
        self._reusables = [Reusable() for _ in range(size)]

    def acquire(self):
        return self._reusables.pop()

    def release(self, reusable):
        self._reusables.append(reusable)


class Reusable:
    """
    Collaborate with other objects for a limited amount of time, then
    they are no longer needed for that collaboration.
    """

    pass


def main():
    reusable_pool = ReusablePool(10)
    reusable = reusable_pool.acquire()
    reusable_pool.release(reusable)


if __name__ == "__main__":
    main()
user_28760790
Related