Optimizing Pandas with Text Analysis

Question

I have been able to create a program that works for a small set of data, however, when I try and scale the amount of data I am working with, my program takes forever. So I was wondering if anyone has a better approach than what follows.

For my simple example, I have a data frame with various sentences people have written where there is a lot of variation in the formatting of the text. Sometimes there may be spaces to separate the words, however sometimes it may just be one long string without any spaces. In addition, there can be misspells, but I am ignoring those.

Here are some examples of what I am talking about

Name	Sentence
John	"This is my first sentence"
Jane	"tHisIsMyFirstsEntenceToo!"
Bob	"Canyoube1ievethis?"
Anna	"Why are we doing this first?"

This is a reduced set of data that I am working with, but what I am trying to do is find the largest word in the string from a list of strings. So for this example, here is the list of words that I want to look for in these strings and identify the longest word in that was found ["sentence", "this", "first", "believe"]

The output with the data and list should be

Name	Sentence	Longest Word
John	"This is my first sentence"	"sentence"
Jane	"tHisIsMyFirstsEntenceToo!"	"sentence"
Bob	"Canyoube1ievethis?"	"this"
Anna	"Why are we doing this first?"	"first"

Obviously the first thing to do is standardize the text, so I lowercase the sentences and the words in the list are already lowercase, but I will include that too just for the sake of generalizing.

import pandas as pd
import numpy as np
words = ["sentence", "this", "first", "believe"]
df['sentence_lower'] = df['Sentence'].str.lower()
words = [word.lower() for word in words]

From here, I do not know what the best approach is to get the desired output. I just iterated over the words and checked to see if the word was in the sentence and then see if it is longer than the current match.

df['Longest Word'] = ''
for word in words:
     df['Longest Word'] = np.where((df['sentence_lower'].str.contains(word)) & (
                                    df['Longest Word'].str.len() < len(word)),
                                         word, df['Longest Word'])

This does work, but it is pretty slow. Is there a better way of doing this?

userCDbL · Answer

With the examples you provided:

import pandas as pd

df = pd.DataFrame(
    {
        "Name": ["John", "Jane", "Bob", "Anna"],
        "Sentence": [
            "This is my first sentence",
            "tHisIsMyFirstsEntenceToo!",
            "Canyoube1ievethis?",
            "Why are we doing this first?",
        ],
    }
)

words = ["sentence", "this", "first", "believe"]

Here is another way to do it:

words = sorted(set(words), key=len)
df["Longest Words"] = df["Sentence"].apply(
    lambda x: max([word if word in x.lower() else "" for word in words], key=len)
    )

print(df)
# Output
   Name                      Sentence Longest Words
0  John     This is my first sentence      sentence
1  Jane     tHisIsMyFirstsEntenceToo!      sentence
2   Bob            Canyoube1ievethis?          this
3  Anna  Why are we doing this first?         first

On my machine, it takes 0.00011 seconds in average (50,000 runs), compared to 0.00210 seconds for your code, which is nearly 20 times faster.

Optimizing Pandas with Text Analysis

1 Answers