Fastest way to get word counts from text using search words from lists?

user_3225420 0
Viewed 188

I'm actually trying to make a rudimentary classifier so I'd be OK with an NLTK solution but my first couple of attempts at this have been with Pandas.

I have a couple of lists that I want to check the text for and get word counts on, and then return an ordered 

import pandas as pd
import re
fruit_sentences = ["Monday: Yellow makes me happy.  So I eat a long, sweet fruit with a peel.",
                                "Tuesday: A fruit round red fruit with a green leaf a day keeps the doctor away.",
                                "Wednesday: The stout, sweet green fruit keeps me on my toes!",
                                "Thursday: Another day with the red round fruit.  I like to keep the green leaf.",
                                "Friday: Long yellow fruit day, peel it and it's ready to go."]
df = pd.DataFrame(fruit_sentences, columns = ['text'])
banana_words = ['yellow', 'long', 'peel']
apple_words = ['round', 'red', 'green leaf']
pear_words = ['stout', 'sweet', 'green']

print(df['text'].str.count(r'[XYZ_word in word list]'))

Here is where the code blows up because the str.count() does not accept a list.

The end goal is to have a returned list of tuples like this:

fruits = [('banana', 5), ('pear', 6), ('apple', 6)]

Yes, I could iterate over all the lists to do this but it seems like I just don't know enough Python rather than Python doesn't know how to handle this elegantly.

I found this question but it looks like everyone answered it incorrectly or with a different solution than what was actually asked for, it's here.

Thank you for helping this newbie figure it out!

pandas nltk python-3.x python
4 Answers

I think need:

#create dict for names of lists
d = {'banana': banana_words, 'apple': apple_words, 'pear':pear_words}
#create one big list
L =  ' '.join(df['text'])

#count each value of lists and sum in generator
out = [(k, sum(L.count(x) for x in v)) for k,v in d.items()]
print (out)

[('banana', 4), ('apple', 6), ('pear', 6)]

If want check lowercase values:

#create one big list
L =  ' '.join(df['text']).lower()

#count each value of lists and sum in generator
out = [(k, sum(L.count(x) for x in v)) for k,v in d.items()]
print (out)

[('banana', 6), ('apple', 6), ('pear', 6)]
user_29010020

Use str.contains with regular expression.

# store lists in a dictionary for checking values.
a = {'banana': banana_words, 'apple': apple_words, 'pear':pear_words}

d = {}
# regular expression to match words
regex = '(?<!\S){0}[^\w\s]?(?!\S)'  

for i, j in a.items():
    d[i] = sum([df['text'].str.contains(regex.format(k), case=False).sum() for k in j])

print (d.items())

Output:

[('banana', 6), ('apple', 6), ('pear', 6)]
user_99488270

How about:

python 3.6.4 / pandas 0.23.4:

import pandas as pd

def count(word_list):
    d = pd.Series(word_list).apply(lambda x: s.str.count(x))
    return d.sum()

fruit_sentences = ["Monday: Yellow makes me happy.  So I eat a long, sweet 
fruit with a peel.",
                        "Tuesday: A fruit round red fruit with a green leaf a day keeps the doctor away.",
                        "Wednesday: The stout, sweet green fruit keeps me on my toes!",
                        "Thursday: Another day with the red round fruit.  I like to keep the green leaf.",
                        "Friday: Long yellow fruit day, peel it and it's ready to go."]

banana_words = ['yellow', 'long', 'peel']
apple_words = ['round', 'red', 'green leaf']
pear_words = ['stout', 'sweet', 'green']

keywords = {'banana': banana_words, 'apple': apple_words, 'pear': pear_words}

s = pd.Series(fruit_sentences)
res = pd.DataFrame(columns=[])
res['type'] = pd.Series(list(keywords.keys()))
res['value'] = pd.Series(list(keywords.values())).apply(lambda x: count(x)).sum(axis=1)
print(list(res.itertuples(index=False, name=None)))

python 2.7.11 / pandas 0.17:

import pandas as pd


def count(word_list):
    d = pd.Series(word_list).apply(lambda x: s.str.count(x))
    return d.sum()


fruit_sentences = ["Monday: Yellow makes me happy.  So I eat a long, sweet fruit with a peel.",
                        "Tuesday: A fruit round red fruit with a green leaf a day keeps the doctor away.",
                        "Wednesday: The stout, sweet green fruit keeps me on my toes!",
                        "Thursday: Another day with the red round fruit.  I like to keep the green leaf.",
                        "Friday: Long yellow fruit day, peel it and it's ready to go."]

banana_words = ['yellow', 'long', 'peel']
apple_words = ['round', 'red', 'green leaf']
pear_words = ['stout', 'sweet', 'green']

keywords = {'banana': banana_words, 'apple': apple_words, 'pear': pear_words}

s = pd.Series(fruit_sentences)

res = pd.DataFrame(columns=[])
res['type'] = pd.Series(keywords.keys())
res['value'] = pd.Series(keywords.values()).apply(lambda x: count(x)).sum(axis=1)

print(list(res.itertuples(index=False)))

both will give you:

[('banana', 4), ('apple', 6), ('pear', 6)]
user_90206660

For this, I would use a dictionary look up (super fast), and use Counter O(n) to create the dict.

# create a dict of look up values
d = {'banana': banana_words, 'apple': apple_words, 'pear':pear_words}

# preprocess data
df['text'] = df['text'].str.lower()
df['text'] = [re.sub(r'[^a-zA-Z0-9\s]','',x) for x in df['text']]
df['text'] = df.text.str.split()

# flatten the list and create a dict
from collections import Counter 

my_list = [i for s in df['text'] for i in s]
word_count = Counter(my_list)

# final job
output_dict = {k:len([x for x in v if x in word_count]) for k,v in d.items()}
sorted(output_dict.items(), key=lambda x: x[1])

[('apple', 2), ('banana', 3), ('pear', 3)]
user_92992590
Related