'>' not supported between instances of 'str' and 'int' pandas function for getting threshold

user_16276223 0
Viewed 81

I have a df

import pandas as pd
df= pd.DataFrame({'ID': [1,2,3], 
                  
                           'Text':['This num dogs and cats is (111)888-8780 and other',
                              'dont block cow 23 here',
                              'cat two num: dog  and cows here'], 
                                   
                                  'Match':[[('cats', 86), ('dogs', 86), ('dogs', 29)],
                                   [('cow', 33), ('dont', 57), ('cow', 100)], 
                                             [('cat', 100), ('dog', 100), ('cows', 86)] ]
                      })

And it looks like this

    ID                       Text                                   Match
0   1   This num dogs and cats is (111)888-8780 and other   [(cats, 86), (dogs, 86), (dogs, 29)]
1   2   dont block cow 23 here                              [(cow, 33), (dont, 57), (cow, 100)]
2   3   cat two num: dog and cows here                      [(cat, 100), (dog, 100), (cows, 86)]

My goal is to create a function that only keeps certain item within Match column that are above a certain threshold (e.g. 80) so I tried the following

def threshold(column):
    
    column_tup = column
    
    keep_tuple = []
    
    for col in column_tup:
        if column_tup > 80:
            keep_tuple.append()
    
    return pd.Series([keep_tuple], index = ['Keep_Words'])

df_thresh = df.join(df.apply(lambda x: threshold(x),  axis = 1))

But this gives me an error

'>' not supported between instances of 'str' and 'int'

My goal is to get a df with a new column Keep_Words that looks like the following where only score above 85 are kept in Keep_Words

     ID    Text    Match   Keep_Words
0   1      data    data   [(cats, 86), (dogs, 86)]              
1   2      data    data   [(cow, 100)]
2   3      data    data   [(cat, 100), (dog, 100)]

How do I alter my code to reach my goal?

pandas tuples list function python
1 Answers

Since you're trying to change only the Match column, you might as well only pass that column to apply:

df.Match.apply(threshold)

where we don't use axis argument anymore since it is a Series we are applying over and it has only one axis anyway.

Then, each time your function is called, a row of df.Match will be passed and get assigned to the function argument, so we can rename the function signature to:

def threshold(match_row):

for readability.

So, match_row will be a list, e.g., in the first turn it'll be [(cats, 86), (dogs, 86), (dogs, 29)]. We can iterate over as you did but with 2 for-loop variable as:

for name, val in match_row:

so that name will become the first entry of each tuple and val is the second. Now we can do the filtering:

keep_tuple = []
for name, val in match_row:
    if val > 85:
        keep_tuple.append((name, val))

which is fine but not so Pythonic because there is list comprehensions:

keep_tuple = [(name, val) for name, val in match_row if val > 85]

Lastly we can return this as you did:

return pd.Series([keep_tuple], index=["Keep_Words"])

As for calling and assignment, we can join as you did:

df_thresh = df.join(df.Match.apply(threshold))

All in all,

def threshold(match_row):
    keep_tuple = [(name, val) for name, val in match_row if val > 85]
    return pd.Series([keep_tuple], index=["Keep_Words"])

df_thresh = df.join(df.Match.apply(threshold))

which gives

>>> df_thresh

   ID                                               Text                                 Match                            Keep_Words
0   1  This num dogs and cats is (111)888-8780 and other  [(cats, 86), (dogs, 86), (dogs, 29)]              [(cats, 86), (dogs, 86)]
1   2                             dont block cow 23 here   [(cow, 33), (dont, 57), (cow, 100)]                          [(cow, 100)]
2   3                    cat two num: dog  and cows here  [(cat, 100), (dog, 100), (cows, 86)]  [(cat, 100), (dog, 100), (cows, 86)]

Lastly, for the error you got: I didn't get that error but the infamous

ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

error, which was because of this line

if column_tup > 80:

where column_tup is a whole row as a pd.Series but its behaviour in boolean context is ambiguous.

user_93321870
Related