Threads is not executing in parallel python with ThreadPoolExecutor

user_11051405 0
Viewed 4124

I'm new in python threading and I'm experimenting this: When I run something in threads (whenever I print outputs), it never seems to be running in parallel. Also, my functions take the same time that before using the library concurrent.futures (ThreadPoolExecutor). I have to calculate the gains of some attributes over a dataset (I cannot use libraries). Since I have about 1024 attributes and the function was taking about a minute to execute (and I have to use it in a for iteration) I dicided to split the array of attributes into 10 (just as an example) and run the separete function gain(attribute) separetly for each sub array. So I did the following (avoiding some extra unnecessary code):

def calculate_gains(self):
    splited_attributes = np.array_split(self.attributes, 10)
    result = {}
    for atts in splited_attributes:
        with concurrent.futures.ThreadPoolExecutor() as executor:
            future = executor.submit(self.calculate_gains_helper, atts)
            return_value = future.result()
            self.gains = {**self.gains, **return_value}

Here's the calculate_gains_helper:

def calculate_gains_helper(self, attributes):
    inter_result = {}
    for attribute in attributes:
        inter_result[attribute] = self.gain(attribute)
    return inter_result

Am I doing something wrong? I read some other older posts but I couldn't get any info. Thanks a lot for any help!

python-multithreading machine-learning concurrency python
2 Answers

Python threads do not run in parallel (at least in CPython implementation) because of the GIL. Use processes and ProcessPoolExecutor to really have parallelism

with concurrent.futures.ProcessPoolExecutor() as executor:
    ...
user_18122620

You submit and then wait for each work item serially so all the threads do is slow everything down. I can't guarantee this will speed things up much because you are still dealing with the python GIL that keeps python level stuff from working in parallel, but here goes.

I've created a thread pool and pushed everything possible into the worker, including the slicing of self.attributes. 

def calculate_gains(self):
    with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
        result_list = executor.map(self.calculate_gains_helper,
            ((i, i+10) for i in range(0, len(self.attributes), 10)))
    for return_value in result_list:
        self.gains = {**self.gains, **return_value}

def calculate_gains_helper(self, start_end):
    start, end = start_end
    inter_result = {}
    for attribute in self.attributes[start:end]:
        inter_result[attribute] = self.gain(attribute)
    return inter_result
user_6420700
Related