Python Concurrent.Futures ProcessPool Executor wrong output

user_16464260 0
Viewed 113

I am trying to use the ProcessPoolExecutor() to run some functions but I cant manage to understand how to get the return of the functions out of the with.

 def threaded_upload(i):
    time.sleep(2)
    if i == 0:
        k = 10
    elif i == 2:
        k = i*i
    else:
        k = -99
    return [k]

 def controller():
    if __name__ == "__main__":
        futures = []
        with ProcessPoolExecutor() as pool:
            for paso in range(4):
                futuro_i = pool.submit(threaded_upload,paso)
                wth=[futuro_i.result()]
                futures.append(futuro_i)

            wait(futures, return_when=ALL_COMPLETED)

            merged_list = []
            for future in futures:
                for valor in future.result():
                    merged_list.append(valor)
            Lista_Final = merged_list
            wait(futures, return_when=ALL_COMPLETED)
        return Lista_Final

  print(controller())

The output of the code is: None [10, -99, 4, -99]

I am not sure why? The "wait" doesn't seem to wait until all functions are executed either.

To be honest, I have been reading and reading for a few days but the description of concurrent.futures or multiprocessing are more advanced that my current knowledge.

Any clarification will be appreciated. Thanks in advance.

concurrent.futures submit wait python
1 Answers

You first submit the jobs and then wait for the results. You can also return an integer instead of a list and then skip the inner loop:

test.py:

import random
import time

from concurrent.futures import ProcessPoolExecutor, wait


def worker(i):
    t = random.uniform(1, 5)

    print(f"START: {i} ({t:.3f}s)")

    time.sleep(t)

    if i == 0:
        k = 10
    elif i == 2:
        k = i * i
    else:
        k = -99

    print(f"END: {i}")

    return k


def main():
    futures = []

    with ProcessPoolExecutor() as pool:
        for i in range(4):
            future = pool.submit(worker, i)
            futures.append(future)

        results = []

        done, pending = wait(futures) # ALL_COMPLETED is the default value

        for future in done:
            results.append(future.result())

        print(results)


if __name__ == "__main__":
    main()

Test:

$ python test.py
START: 0 (1.608s)
START: 1 (1.718s)
START: 2 (1.545s)
START: 3 (1.588s)
END: 2
END: 3
END: 0
END: 1
[10, -99, 4, -99]
user_14166720
Related