Running a `ThreadPoolExecutor` only until a number of results have been returned

An old man in the sea. 0
Viewed 33

In my code, I have the following lines:

with ThreadPoolExecutor() as pool:
            for results in pool.map(load_bucket, buckets):
                 res_list.append(results)

I'm trying to break the map, as soon as the list res_list reaches a certain length. I've tried the following:

with ThreadPoolExecutor() as pool:
            for results in pool.map(load_bucket, buckets):
                if len(res_list)- bucket_size < number_reviews:
                    res_list.append(results)
                else:
                    break

This doesn't seem to work.

concurrent.futures python
1 Answers

the best option here is to use ThreadPoolExecutor.submit instead of map as map will construct the entire output as a list before starting the loop, while submit will return feature, and later convert the returned futures to the actual output after the loop is done, note that if you don't want to block the main thread waiting for the second loop to finish then you can use run this inside AsyncIO loop as it is easily converted to awaitable function (or just run this code in another thread).

from concurrent.futures import ThreadPoolExecutor
from concurrent.futures import as_completed

def do_work(input_val):
    print(input_val)
    return input_val

if __name__ == "__main__":
    buckets = list(range(10))
    res_list = []
    actual_results = []
    with ThreadPoolExecutor() as pool:
        for entry in buckets:
            if len(res_list)  < 5:
                res_list.append(pool.submit(do_work,entry)) # returns futures ,not actual return
            else:
                break

        # futures must be converted to the actual return
        # output is in finish order, not submission order
        # for submission order remove the 'as_completed'
        for entry in as_completed(res_list):
            actual_results.append(entry.result())

    print(actual_results)

and the result is (out of order):

0
1
2
3
4
[2, 0, 1, 3, 4]
Ahmed AEK0
Related