iterating over a pandas dataframe super fast

user_17567185 0
Viewed 177

I am trying to create labels for my dataset which is just a csv file with 20_000 lines of text. I 've used the following method but it takes about 15 minutes to create all the results. The desired outcome is

"This was a really crappy product", generated_label_with_confidence_score

import pandas as pd
import flair
from flair.models import TextClassifier
from flair.data import Sentence
import numba
import tqdm
import numpy as np

#@numba.jit
def predict_label(text):
  sentence = Sentence(text)
  classifier.predict(sentence)
  # print sentence with predicted labels
  return sentence.labels

for index, row in tqdm.tqdm(df_train.iterrows()):
    df_train.iloc[index, 1] = predict_label(df_train.iloc[index,0])
flair numba pandas python
1 Answers

Instead of iterating over the dataframe using for loop, you can use df.apply to apply a function on each entry of a column. Read its official documentation. https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.apply.html

I have tested time taken both by loop and df.apply here is a comparison.

In the following piece of code I applied a function(returning length of string) to text column to a dataframe having 20,000 entries.

def len_text(text):
    return len(text)

df_train = pd.DataFrame(['example sentence']*20000, columns = ['text'])

for index, row in tqdm.tqdm(df_train.iterrows()):
    df_train.iloc[index, 0] = len_text(df_train.iloc[index,0])

The above code took approximately 5 seconds to execute. Let's see a better version by using df.apply and see how much time it will take.

df_train = pd.DataFrame(['example sentence']*20000, columns = ['text'])
st_time = time.time()

df_train['text'] = df_train['text'].apply(len_text)

en_time = time.time()
print(en_time - st_time)

The code in the cell above with df.apply took only 0.027 seconds to run on 20.000 entries.

So using df.apply instead of looping over the dataframe improved time by 4.973 seconds out of 5 seconds.

user_122081290
Related