How do I get tqdm working on pandas apply?

user_3737135 0
Viewed 2051

Tqdm documentation shows an example of tqdm working on pandas apply using progress_apply. I adapted the following code from here https://tqdm.github.io/docs/tqdm/ on a process that regularly take several minutes to perform (func1 is a regex function).

from tqdm import tqdm
tqdm.pandas()
df.progress_apply(lambda x: func1(x.textbody), axis=1)

The resulting progress bar doesn't show any progress. It just jumps from 0 at the start of the loop to 100 when it is finished. I am currently running tqdm version 4.61.2

tqdm pandas python
1 Answers

Utilizing tqdm with pandas

Generally speaking, people tend to use lambdas when performing operations on a column or row. This can be done in a number of ways.

  • Please note: that if you are working in jupyter notebook you should use tqdm_notebook instead of tqdm.
  • Also I'm not sure what your code looks like but if you're simply following the example given in the tqdm docs, and you're only performing 100 interations, computers are fast and will blow through that before your progress bar has time to update. Perhaps it would be more instructive to use a larger dataset like I provided below.

Example 1:

from tqdm import tqdm # version 4.62.2
import pandas as pd # version 1.4.1
import numpy as np

tqdm.pandas(desc='My bar!') # lots of cool paramiters you can pass here. 
# the below line generates a very large dataset for us to work with. 
df = pd.DataFrame(np.random.randn(100000000, 4), columns=['a','b','c','d'])
# the below line will square the contents of each element in an column-wise 
# fashion 
df.progress_apply(lambda x: x**2)

Output:

Output

Example 2:

# you could apply a function within the lambda expression for more complex 
# operations. And keeping with the above example... 

tqdm.pandas(desc='My bar!') # lots of cool paramiters you can pass here. 
# the below line generates a very large dataset for us to work with. 
df = pd.DataFrame(np.random.randn(100000000, 4), columns=['a','b','c','d'])

def function(x):
    return x**2
     
df.progress_apply(lambda x: function(x))
user_125143030
Related