Mutiprocessing: use tqdm to show a global progress bar?

Viewed 332

I'm processing on a large pandas dataframe. I split the dataframe into N chunks and process them with a N-process pool.

I hope to display a single global progress bar of the dataframe by row (rather than the process number).

Sample code:

import multiprocessing as mp
import tqdm
import time
import pandas as pd
import numpy as np


def foo(df: pd.DataFrame):
    for row in tqdm.tqdm(df.iterrows(), total=df.shape[0]):
        time.sleep(0.1)
        # process on row
    return df


df = pd.DataFrame(np.random.randint(0,100,size=(100, 4)), columns=list('ABCD'))
chunks = np.array_split(df, 8)
pool = mp.Pool(processes=8)
res = []

for i in range(8):
    res.append(pool.apply_async(foo, args=(chunks[i],)))

pool.close()
pool.join()
df_res = pd.concat([i.get() for i in res])

The expected result would be something like:

 58%|█████▊    | 58/100 [00:03<00:02,  9.87it/s]

The inaccurate ETA would be fine. Any help or suggestions are welcome.

0 Answers
Related