concurrent.futures.ThreadPoolExecutor unreleased memory

Viewed 152

I am currently using concurrent.futures.ThreadPoolExecutor in my code to execute one of my code sections concurrently but after running it for a while, I can see a raise in memory consumption of my main process which just keep rising and doesn't get freed.

I'm currently using the concurrent.futures.ThreadPoolExecutor like that:

with concurrent.futures.ThreadPoolExecutor(max_workers=10) as pool:
        processing_data = pool.map(handle_item, items_dict.values())

# Using the data in processing_data...

I've seen some other people having issues the memory consumption using this module but other questions on SO that I've found were all using it a bit differently and It felt like it is not relevant for my case.

If someone knows what I am doing wrong here or how I might fix that, I'll be grateful

Thanks everybody

Edit:

Here is an example code that reproduces the potential memory leak:

#!/usr/bin/env python3

import os
import time
import psutil
import random
import concurrent.futures

from memory_profiler import profile as mem_profile

p = psutil.Process(os.getpid())

def do_magic(values):
    return None

@mem_profile
def foo():
    a = {i: chr(i) for i in range(1024)}
    with concurrent.futures.ThreadPoolExecutor(max_workers=10) as pool:
        proccessed_data = pool.map(do_magic, [1,2,3,4,5,6,7,8,9,10])

def fooer():
    while True:
        foo()
        time.sleep(1)

fooer()
0 Answers
Related