Check to see if column values exist in dictionary [pandas]

user_8643941 0
Viewed 1618

Can a data frame column (Series) of lists be used as a conditional check within a dictionary?

I have a column of lists of words (split up tweets) that I'd like to feed to a vocab dictionary to see if they all exist - if one does not exist, I'd like to skip it, continue on and then run a function over the existing words.

This code produces the intended result for one row in the column, however, I get a "unhashable type list" error if I try to apply it to more than one column.

w2v_sum = w2v[[x for x in train['words'].values[1] if x in w2v.vocab]].sum()

Edit with reproducible example:

df = pd.DataFrame(data={'words':[['cow','bird','cat'],['red','blue','green'],['low','high','med']]})

d = {'cow':1,'bird':4,'red':1,'blue':1,'green':1,'high':6,'med':3}

Desired output is total (sum of the words within dictionary):

total   words
0   5   [cow, bird, cat]
1   3   [red, blue, green]
2   9   [low, high, med]
pandas dictionary python
2 Answers

This should do what you want:

import pandas as pd
df = pd.DataFrame(data={'words':[['cow','bird','cat'],['red','blue','green'],['low','high','med']]})

d = {'cow':1,'bird':4,'red':1,'blue':1,'green':1,'high':6,'med':3}

EDIT:

To reflect the lists inside the column, see this nested comprehension:

list_totals = [[d[x] for x in y if x in d] for y in df['words'].values]
list_totals = [sum(x) for x in list_totals]
list_totals
[5, 3, 9]

You can then add list_totals as a column to your pd.

user_105264410

One solution is to use collections.Counter and a list comprehension:

from collections import Counter

d = Counter({'cow':1,'bird':4,'red':1,'blue':1,'green':1,'high':6,'med':3})

df['total'] = [sum(map(d.__getitem__, L)) for L in df['words']]

print(df)

                words  total
0    [cow, bird, cat]      5
1  [red, blue, green]      3
2    [low, high, med]      9

Alternatively, if you always have a fixed number of words, you can split into multiple series and use pd.DataFrame.applymap:

df['total'] = pd.DataFrame(df['words'].tolist()).applymap(d.get).sum(1).astype(int)
user_92095460
Related