Similar to "Generate n-grams from Pandas column while persisting another column" (unsolved), but with values

user_14395605 0
Viewed 69

I have some lines of text, and then their relevance weight.

Weight, Text
10, "I like apples"
20, "Someone needs apples"

Is it possible to get the combinations, keeping the values in the weights column? Something like:

weight, combinations
10, [I like]
10, [I apples]
10, [like apples]
20, [someone needs]
20, [someone apples]
20, [needs apples]

"Generate n-grams from Pandas column while persisting another column" (unsolved) is a similar question, but it is unsolved.

Thanks!!!

pandas n-gram python
2 Answers
from itertools import combinations
import pandas as pd

df = pd.DataFrame({'Weight': [10, 20],
                   'Text': ["I like apples", "Someone needs apples"]})
df['Combinations'] = df.Text.apply(lambda x : list(combinations(x.split(), 2)))
df = df.explode('Combinations')
df.drop('Text', axis=1, inplace=True)

print(df)

Output:

   Weight       Combinations
0      10          (I, like)
0      10        (I, apples)
0      10     (like, apples)
1      20   (Someone, needs)
1      20  (Someone, apples)
1      20    (needs, apples)
user_14535080

Panda's explode() is a relatively new feature (.25) that enables an efficient solution like David M.'s. It also addresses issues that previous typical solutions had when the list column was a list of lists.

Before explode() a typical solution would look like this:

someDF = pd.DataFrame({col:np.repeat(someListColDF[col].values,    
                      someListColDF[someListCol].str.len()) for col in someListColDF.columns.drop(someListCol)} ).assign(**{someListCol:np.concatenate(someListColDF[someListCol].values)})[someListColDF.columns]

But this doesn't seem to work when someListCol is a list of lists.

Here are the intermediate steps:

import itertools
someDF = pd.DataFrame([[10, "I like apples"],[20, "Someone needs apples"]], 
                     columns = ["Weight", "Text"])

Repeat the Weights the correct number of times

weightList = np.repeat(someDF["Weight"].values, someDF["Permute"].str.len())

Get all possible 2 element permutations (assuming order matters), and concatenate them into an array

someDF["Permute"] = someDF["Text"].apply(lambda x: list((itertools.permutations(x.split(),2))))

print(someDF["Permute"])

0    [(I, like), (I, apples), (like, I), (like, app...
1    [(Someone, needs), (Someone, apples), (needs, ...

permuteList = np.concatenate(someDF["Permute"].values)
print(permuteList)

array([['I', 'like'],
   ['I', 'apples'],
   ['like', 'I'],
   ['like', 'apples'],
   ['apples', 'I'],
   ['apples', 'like'],
   ['Someone', 'needs'],
   ['Someone', 'apples'],
   ['needs', 'Someone'],
   ['needs', 'apples'],
   ['apples', 'Someone'],
   ['apples', 'needs']], dtype='<U7')

But when I try to glue these together in normal ways by using, for example, np.column_stack(), np.vstack() and np.concatenate(axis=1), the list of lists keeps getting misinterpreted, and reshaping doesn't seem to help.

Ultimately I had to resort to this kludge:

newDF = pd.DataFrame(weightList, columns=["Weight"])
newDF["Permute"] = [i for i in permuteList]

Output

    Weight            Permute
0       10          [I, like]
1       10        [I, apples]
2       10          [like, I]
3       10     [like, apples]
4       10        [apples, I]
5       10     [apples, like]
6       20   [Someone, needs]
7       20  [Someone, apples]
8       20   [needs, Someone]
9       20    [needs, apples]
10      20  [apples, Someone]
11      20    [apples, needs]

All this is a roundabout way of saying Thank You to the Pandas developers for giving us explode()!

user_143567880
Related