Queue.get() memory leak

Viewed 530

I use the following code to produce what looks like a memory leak after emptying a queue with the get() method.

import queue
import os
import psutil

def run(del_after_puts, del_after_gets, n_puts, process):

        mem = queue.Queue()
    
        for msg in range(n_puts):
            msg_put = f'{msg}_0000000000000000000000000000000000000000000000000000000000000333333333333331111111111'
            if msg % 1000000 == 0:
                print(f'puting  {msg} qsize {len(mem.queue)}')
            mem.put(msg_put)
    
    
        print(f'------ put done  ----- qsize {len(mem.queue)}')
        print(f'mem_pct {round(process.memory_percent(), 2)}% ')
    
        if del_after_puts:
            print(f'deleting queue after puts {mem}')
            del mem
            print(f'mem_pct {round(process.memory_percent(), 2)}% ')
            return
    
        for _ in range(n_puts):
            msg_get = mem.get()
            msg = int(msg_get.split('_')[0])
            if msg % 1000000 == 0:
                print(f'getting_q {msg} qsize {len(mem.queue)} ')
            mem.task_done()
            
        print(f'------ gets done  ----- qsize {len(mem.queue)}')
        print(f'mem_pct {round(process.memory_percent(), 2)}% ')
    
        if del_after_gets:
            print(f'deleting queue after gets {mem}')
            del mem
            print(f'mem_pct {round(process.memory_percent(), 2)}% ')
            return
    
    if __name__ == '__main__':
        del_after_puts = False
        del_after_gets = False
        n_puts = 20_000_000
        print()
        print('#########')
        print(f'del_after_puts {del_after_puts} del_after_gets {del_after_gets} n_puts {n_puts}')
    
        process = psutil.Process(os.getpid())
        print('before run')
        print(f'mem_pct {round(process.memory_percent(), 2)}% ')
    
        run(del_after_puts, del_after_gets, n_puts, process)
    
        print(f'after run return')
        print(f'mem_pct {round(process.memory_percent(), 2)}% ')

This script can be run in 3 ways:

  1. Add n_puts elements into the queue and then empty it.
  2. Add n_puts elements into the queue and then delete the queue object
  3. Add n_puts elements into the queue and then empty it and then delete the queue object.

For the 1st and 3rd case, the script seems to produce a memory leak as in the following:

1st case, before putting elements into the queue mem used is 0.15%, after emptying it 2.22%:

#########

del_after_puts False del_after_gets False n_puts 20000000

before run

mem_pct 0.15%

------ put done ----- qsize 20000000

mem_pct 37.61%

------ gets done ----- qsize 0

mem_pct 2.22%

3rd case, before putting elements into the queue mem used is 0.15%, after emptying it 2.22%, after deleting the object, 2.22%:

#########

del_after_puts False del_after_gets True n_puts 20000000

before run

mem_pct 0.15%

------ put done ----- qsize 20000000

mem_pct 37.61%

------ gets done ----- qsize 0

mem_pct 2.22%

deleting queue after gets <queue.Queue object at 0x7fbd87295a10>

mem_pct 2.22%

For the 2nd case, mem_pct at the start is 0.15%, after putting all elements into the queue and just deleting it, 0.16%, which is almost the same.

#########

del_after_puts True del_after_gets False n_puts 20000000

before run

mem_pct 0.15%

------ put done ----- qsize 20000000

mem_pct 37.61%

deleting queue after puts <queue.Queue object at 0x7f29084eca10>

mem_pct 0.16%

As it can be seen, memory returns to the start level only in the second case when only queue.put() is invoked, hence it seems that queue.get() produces a memory leak.

This is persistent across python 3.7, 3.8, as well as 3.9.

I've tried to profile the memory with tracemalloc as well as pympler, but they dont show any leaks at python level, so I suspect that might be a C level leak.

I use queues with threads on applications for logging that are supopsed to run for many weeks, and queues seem to cause leaks and hang my application. I am able to trace leaks over there, and they indicate that seems to come from a deque in a Condition lock used in the queue mechanism, but I've never seen being full of any elements/waiters in my running application, so go figure.

threading.py:348:
waiters_to_notify = _deque(_islice(all_waiters, n))

In any case, is there a way to mitigate and deal with this queue leak? Thanks

1 Answers

If anyone looking for solution, the reason for the leak seem to be the stock python deque used in the Queue implementation. I've changed the deque implementation from the stock one found here:

https://github.com/kata198/python-cllist

Then redefined the Queue class follows:

from cllist import dllist

class DllistQueue(queue.Queue):
    def _init(self, maxsize):
        self.queue = dllist()

Then the results tests results with the new class are:

#########

del_after_puts False del_after_gets True n_puts 20000000

before run

mem_pct 0.15%

puting 0 qsize 0

------ put done ----- qsize 20000000

mem_pct 47.35%

------ gets done ----- qsize 0

mem_pct 0.16%

deleting queue after gets <main.DllistQueue object at 0x7f3a197d8f90>

mem_pct 0.16%

So this linked-listed implementation does not seem to leak, plus it's as fast as the stock one. The only disadvantage is that it uses a little more memory at the top usage.

Related