I'm processing on a large pandas dataframe. I split the dataframe into N chunks and process them with a N-process pool.
I hope to display a single global progress bar of the dataframe by row (rather than the process number).
Sample code:
import multiprocessing as mp
import tqdm
import time
import pandas as pd
import numpy as np
def foo(df: pd.DataFrame):
for row in tqdm.tqdm(df.iterrows(), total=df.shape[0]):
time.sleep(0.1)
# process on row
return df
df = pd.DataFrame(np.random.randint(0,100,size=(100, 4)), columns=list('ABCD'))
chunks = np.array_split(df, 8)
pool = mp.Pool(processes=8)
res = []
for i in range(8):
res.append(pool.apply_async(foo, args=(chunks[i],)))
pool.close()
pool.join()
df_res = pd.concat([i.get() for i in res])
The expected result would be something like:
58%|█████▊ | 58/100 [00:03<00:02, 9.87it/s]
The inaccurate ETA would be fine. Any help or suggestions are welcome.