Fix jumping of multiple progress bars (tqdm) in python multiprocessing

user_6449401 0
Viewed 4846

I want to parallelize a task (progresser()) for a range of input parameters (L). The progress of each task should be monitored by an individual progress bar in the terminal. I'm using the tqdm package for the progress bars. The following code works on my Mac for up to 23 progress bars (L = list(range(23)) and below), but produces chaotic jumping of the progress bars starting at L = list(range(24)). Has anyone an idea how to fix this?

from time import sleep
import random
from tqdm import tqdm
from multiprocessing import Pool, freeze_support, RLock

L = list(range(24)) # works until 23, breaks starting at 24

def progresser(n):
    text = f'#{n}'

    sampling_counts = 10
    with tqdm(total=sampling_counts, desc=text, position=n+1) as pbar:
        for i in range(sampling_counts):
            sleep(random.uniform(0, 1))
            pbar.update(1)

if __name__ == '__main__':
    freeze_support()

    p = Pool(processes=None,
                initargs=(RLock(),), initializer=tqdm.set_lock
                )
    p.map(progresser, L)
    print('\n' * (len(L) + 1))

As an example of how it should look like in general, I provide a screenshot for L = list(range(16)) below.

multiprocessing progess bars

versions: python==3.7.3, tqdm==4.32.1

tqdm progress-bar terminal multiprocessing python
1 Answers

I'm not getting any jumping when I set the size to 30. Maybe you have more processors and can have more workers running.

However, if n grows large you will start to see jumps because of the nature of the chunksize.

I.e p.map will split your input into chunksizes and give each process a chunk. So as n grows larger, so does your chunksize, and so does your ....... yup position (pos=n+1)!

Note: Although map preserves the order of the results returned. The order its computed is arbitrary.

As n grows large I would suggest using processor id as the position to view progress on a per process basis. 

from time import sleep
import random
from tqdm import tqdm
from multiprocessing import Pool, freeze_support, RLock
from multiprocessing import current_process


def progresser(n):
    text = f'#{n}'
    sampling_counts = 10
    current = current_process()
    pos = current._identity[0]-1

    with tqdm(total=sampling_counts, desc=text, position=pos) as pbar:
        for i in range(sampling_counts):
            sleep(random.uniform(0, 1))
            pbar.update(1)

if __name__ == '__main__':
    freeze_support()
    L = list(range(30)) # works until 23, breaks starting at 24
    # p = Pool(processes=None,
    #         initargs=(RLock(),), initializer=tqdm.set_lock
    #         )
    with Pool(initializer=tqdm.set_lock, initargs=(tqdm.get_lock(),)) as p: 
        p.map(progresser, L)
        print('\n' * (len(L) + 1))
user_58908090
Related