I am currently using concurrent.futures.ThreadPoolExecutor in my code to execute one of my code sections concurrently but after running it for a while, I can see a raise in memory consumption of my main process which just keep rising and doesn't get freed.
I'm currently using the concurrent.futures.ThreadPoolExecutor like that:
with concurrent.futures.ThreadPoolExecutor(max_workers=10) as pool:
processing_data = pool.map(handle_item, items_dict.values())
# Using the data in processing_data...
I've seen some other people having issues the memory consumption using this module but other questions on SO that I've found were all using it a bit differently and It felt like it is not relevant for my case.
If someone knows what I am doing wrong here or how I might fix that, I'll be grateful
Thanks everybody
Edit:
Here is an example code that reproduces the potential memory leak:
#!/usr/bin/env python3
import os
import time
import psutil
import random
import concurrent.futures
from memory_profiler import profile as mem_profile
p = psutil.Process(os.getpid())
def do_magic(values):
return None
@mem_profile
def foo():
a = {i: chr(i) for i in range(1024)}
with concurrent.futures.ThreadPoolExecutor(max_workers=10) as pool:
proccessed_data = pool.map(do_magic, [1,2,3,4,5,6,7,8,9,10])
def fooer():
while True:
foo()
time.sleep(1)
fooer()