Creating sentences from pandas dataframe

user_9734248 0
Viewed 440

Suppose I have the following dataset:

    pos   sentence_idx      word
    NNS     1.0            Thousands
    IN      1.0            of
    NNS     1.0            demonstrators
    VBP     1.0            have
    VBN     1.0            marched
... ... ... ...
    PRP     47959.0        they
    VBD     47959.0        responded
    TO      47959.0        to
    DT      47959.0        the
    NN      47959.0        attack

I want to create sentences(for this I have to use sentence_idx). I can do this using the following code:

sent = []
for i in df['sentence_idx'].unique():
  sent.append([(w,t) for w,t in zip(df[df['sentence_idx'] == i]['word'].values.tolist(),df[df['sentence_idx'] == i]['pos'].values.tolist())])

But first of all it's not efficient(using for loops instead of numpy/pandas functions) and it looks ugly. How can I do it more efficiently?

EDIT: The result should be sentences, where each element is a tuple(word, pos):

[[('Thousands', 'NNS'),
  ('of', 'IN'),
  ('demonstrators', 'NNS'),
  ('have', 'VBP'),
  ('marched', 'VBN'),
  ('through', 'IN'),
  ('London', 'NNP'),
  ('to', 'TO'),
  ('protest', 'VB'),
  ('the', 'DT'),
  ('war', 'NN'),
  ('in', 'IN'),
  ('Iraq', 'NNP'),
  ('and', 'CC'),
  ('demand', 'VB'),
  ('withdrawal', 'NN'),
  ('British', 'JJ'),
  ('troops', 'NNS'),
  ('from', 'IN'),
  ('that', 'DT'),
  ('country', 'NN'),
  ('.', '.')],
 [('Families', 'NNS'),
  ('of', 'IN'),
  ('soldiers', 'NNS'),
  ('killed', 'VBN'),
  ('in', 'IN'),
  ('the', 'DT'),
  ('conflict', 'NN'),
  ('joined', 'VBD'),
  ('protesters', 'NNS'),
  ('who', 'WP'),
  ('carried', 'VBD'),
  ('banners', 'NNS'),
  ('with', 'IN'),
  ('such', 'JJ'),
  ('slogans', 'NNS'),
  ('as', 'IN'),
  ('"', '``'),
  ('Bush', 'NNP'),
  ('Number', 'NN'),
  ('One', 'CD'),
  ('Terrorist', 'NN'),
  ('and', 'CC'),
  ('Stop', 'VB'),
  ('Bombings', 'NNS'),
  ('.', '.')],...
pandas python
2 Answers

This should work:

def compute(_):
    return [*zip(_['word'], _['pos'])]

df.groupby('sentence_idx').apply(compute).values.tolist()
user_22059690

Unsure of efficiency, but here are some ways to accomplish this:

df.groupby('sentence_idx')[['word', 'pos']].apply(lambda x: list(zip(*zip(*x.values.tolist())))).tolist()

df.groupby('sentence_idx').apply(lambda x: x[['word', 'pos']].apply(tuple, axis=1).tolist())

df.groupby('sentence_idx').apply(lambda x: [tuple(y) for y in x[['word', 'pos']].values]).tolist()

If you don't necessarily need it as a tuple (i.e. a list will do) it's much simpler:

df.groupby('sentence_idx').apply(lambda x: x[['word', 'pos']].values.tolist()).tolist()
user_91833440
Related