I use the following code to produce what looks like a memory leak after emptying a queue with the get() method.
import queue
import os
import psutil
def run(del_after_puts, del_after_gets, n_puts, process):
mem = queue.Queue()
for msg in range(n_puts):
msg_put = f'{msg}_0000000000000000000000000000000000000000000000000000000000000333333333333331111111111'
if msg % 1000000 == 0:
print(f'puting {msg} qsize {len(mem.queue)}')
mem.put(msg_put)
print(f'------ put done ----- qsize {len(mem.queue)}')
print(f'mem_pct {round(process.memory_percent(), 2)}% ')
if del_after_puts:
print(f'deleting queue after puts {mem}')
del mem
print(f'mem_pct {round(process.memory_percent(), 2)}% ')
return
for _ in range(n_puts):
msg_get = mem.get()
msg = int(msg_get.split('_')[0])
if msg % 1000000 == 0:
print(f'getting_q {msg} qsize {len(mem.queue)} ')
mem.task_done()
print(f'------ gets done ----- qsize {len(mem.queue)}')
print(f'mem_pct {round(process.memory_percent(), 2)}% ')
if del_after_gets:
print(f'deleting queue after gets {mem}')
del mem
print(f'mem_pct {round(process.memory_percent(), 2)}% ')
return
if __name__ == '__main__':
del_after_puts = False
del_after_gets = False
n_puts = 20_000_000
print()
print('#########')
print(f'del_after_puts {del_after_puts} del_after_gets {del_after_gets} n_puts {n_puts}')
process = psutil.Process(os.getpid())
print('before run')
print(f'mem_pct {round(process.memory_percent(), 2)}% ')
run(del_after_puts, del_after_gets, n_puts, process)
print(f'after run return')
print(f'mem_pct {round(process.memory_percent(), 2)}% ')
This script can be run in 3 ways:
- Add n_puts elements into the queue and then empty it.
- Add n_puts elements into the queue and then delete the queue object
- Add n_puts elements into the queue and then empty it and then delete the queue object.
For the 1st and 3rd case, the script seems to produce a memory leak as in the following:
1st case, before putting elements into the queue mem used is 0.15%, after emptying it 2.22%:
#########
del_after_puts False del_after_gets False n_puts 20000000
before run
mem_pct 0.15%
------ put done ----- qsize 20000000
mem_pct 37.61%
------ gets done ----- qsize 0
mem_pct 2.22%
3rd case, before putting elements into the queue mem used is 0.15%, after emptying it 2.22%, after deleting the object, 2.22%:
#########
del_after_puts False del_after_gets True n_puts 20000000
before run
mem_pct 0.15%
------ put done ----- qsize 20000000
mem_pct 37.61%
------ gets done ----- qsize 0
mem_pct 2.22%
deleting queue after gets <queue.Queue object at 0x7fbd87295a10>
mem_pct 2.22%
For the 2nd case, mem_pct at the start is 0.15%, after putting all elements into the queue and just deleting it, 0.16%, which is almost the same.
#########
del_after_puts True del_after_gets False n_puts 20000000
before run
mem_pct 0.15%
------ put done ----- qsize 20000000
mem_pct 37.61%
deleting queue after puts <queue.Queue object at 0x7f29084eca10>
mem_pct 0.16%
As it can be seen, memory returns to the start level only in the second case when only queue.put() is invoked, hence it seems that queue.get() produces a memory leak.
This is persistent across python 3.7, 3.8, as well as 3.9.
I've tried to profile the memory with tracemalloc as well as pympler, but they dont show any leaks at python level, so I suspect that might be a C level leak.
I use queues with threads on applications for logging that are supopsed to run for many weeks, and queues seem to cause leaks and hang my application. I am able to trace leaks over there, and they indicate that seems to come from a deque in a Condition lock used in the queue mechanism, but I've never seen being full of any elements/waiters in my running application, so go figure.
threading.py:348:
waiters_to_notify = _deque(_islice(all_waiters, n))
In any case, is there a way to mitigate and deal with this queue leak? Thanks