Sentence are tokenized to words in pandas but the indexes are from 0 till length of first sentence and again for second sentence it starts from 0

user_7216834 0
Viewed 36

I have a python df,

     DF1 = Index  Words    Tags
             0      I       O
             1     Love     B
             2     India    I
             0    Flowers   B
             1      are     O
             2   Beautiful  B

I want the output to be,

   DF1 = Index  Words    Tags   Sent
             0      I       O    1
             1     Love     B    1
             2     India    I    1
             0    Flowers   B    2
             1      are     O    2
             2   Beautiful  B    2

I would want to get a df

python
2 Answers

try this, pandas.DataFrame.Shift to create a boolean mask & apply pandas.Series.cumsum.

df['Sent'] = (df.Index - df.Index.shift()).lt(0).cumsum() + 1

   Index      Words Tags  Sent
0      0          I    O     1
1      1       Love    B     1
2      2      India    I     1
3      0    Flowers    B     2
4      1        are    O     2
5      2  Beautiful    B     2
user_49850990

You need to tokenize the whole sentence. Concat both first.

user_89806340
Related