Find elements between words in an array in dataframe Pandas

user_15112773 0
Viewed 47

I have a dataframe:

import pandas as pd
data = {'token_1': [['cat', 'bag', 'sitting'],
                    ['dog', 'eats', 'bowls'],
                    ['mouse', 'mustache', 'tail'],
                   ['dog', 'eat', 'meat']],
        'token_2': [['cat', 'from', 'bag', 'cat', 'in', 'bag', 'sitting', 'whole', 'day'],
                    ['dog', 'eats', 'from', 'bowls', 'dog', 'eats', 'always', 'from', 'bowls', 'eats', 'bowl'],
                   ['mouse', 'with', 'a', 'big', 'tail', 'and,' 'ears', 'a', 'mouse', 'with', 'a', 'mustache', 'and', 'a', 'tail' ,'runs', 'fast'],
                   ['dog', 'eat', 'meat', 'chicken', 'from', 'bowl','dog','see','meat','eat']]}

df = pd.DataFrame(data)

I have no conjunctions and prepositions in the token_1 column. I would like to find them from the token_2 column.That is, find the words between the crossed tokens. As far as I understand there are several steps:

  1. check the first intersection of token_1[0]
  2. look if the next word is less than the length of 4, if yes then add it in list. If not, then go to the first intersection of token_1[1]
  3. Again we look if the next word is less than length 4
  4. Repeat the process until we reach the last token_1[2]
  5. If there is nothing between the tokens, then return them

Or is there some simpler method?Ultimately, I would like to get a new_token column:

+-----------------------+---------------------------------+--------------------------------------------------------------------------------------------+
|token_1                |new_tokens                       |token_2                                                                                     |
+-----------------------+---------------------------------+--------------------------------------------------------------------------------------------+
|[cat, bag, sitting]    |[cat, in, bag, sitting]          |[cat, from, bag, cat, in, bag, sitting, whole, day]                                         |
|[dog, eats, bowls]     |[dog, eats, from, bowls]         |[dog, eats, from, bowls, dog, eats, always, from, bowls, eats, bowl]                        |
|[mouse, mustache, tail]|[mouse, with,mustache, and, tail]|[mouse, with, a, big, tail, and,ears, a, mouse, with, a, mustache, and, a, tail, runs, fast]|
|[dog, eat, meat]       |[dog, eat, meat]                 |[dog, eat, meat, chicken, from, bowl, dog, see, meat, eat]                                  |
+-----------------------+---------------------------------+--------------------------------------------------------------------------------------------+

Thanks for the help!

pandas dataframe python
1 Answers

use set intersection and pandas series apply

data = {'token_1': [['cat', 'bag', 'sitting'],
                    ['dog', 'eats', 'bowls'],
                    ['mouse', 'mustache', 'tail'],
                   ['dog', 'eat', 'meat']],
        'token_2': [['cat', 'from', 'bag', 'cat', 'in', 'bag', 'sitting', 'whole', 'day'],
                    ['dog', 'eats', 'from', 'bowls', 'dog', 'eats', 'always', 'from', 'bowls', 'eats', 'bowl'],
                   ['mouse', 'with', 'a', 'big', 'tail', 'and,' 'ears', 'a', 'mouse', 'with', 'a', 'mustache', 'and', 'a', 'tail' ,'runs', 'fast'],
                   ['dog', 'eat', 'meat', 'chicken', 'from', 'bowl','dog','see','meat','eat']]}

df = pd.DataFrame(data)
df.reset_index(inplace=True)

df['intersect']=df.apply(lambda x: set(x['token_1']).intersection(set(x['token_2'])),axis=1)
print(df)

def ConvertToDict(lst):
    res_dct = {i: lst[i] for i in range(0, len(lst), 1)}
    return res_dct

df['new_result']=''
for index,row in df.iterrows():
    #print(row['token_2'])
    dct2=ConvertToDict(row['token_2'])
    dct1=ConvertToDict(list(row['intersect']))
    #print(dct)
    #print(tokens)
    lst=[]
    for token in dct2.items():
        lst.append(token[1])
        #print("token",token[1])
        for i in range(len(dct2)):
            if dct2.get(i)==token: 
                if i<len(dct):
                    word=dct2.get(i+1)
                    if len(word)<3:
                        lst.append(word)
                    break
        #print(lst)
    df.at[index,"new_result"]=lst
    
print(df)

output:

    token_2                intersect  \
0  [cat, from, bag, cat, in, bag, sitting, whole,...      {sitting, cat, bag}   
1  [dog, eats, from, bowls, dog, eats, always, fr...       {dog, bowls, eats}   
2  [mouse, with, a, big, tail, and,ears, a, mouse...  {tail, mouse, mustache}   
3  [dog, eat, meat, chicken, from, bowl, dog, see...         {dog, meat, eat}   

                                          new_result  
0  [cat, from, bag, cat, in, bag, sitting, whole,...  
1  [dog, eats, from, bowls, dog, eats, always, fr...  
2  [mouse, with, a, big, tail, and,ears, a, mouse...  
3  [dog, eat, meat, chicken, from, bowl, dog, see...
user_40011770
Related