How to get sentence number in spaCy?

user_943340 0
Viewed 3977

I get the tokens for a string as

doc = nlp(u"This is the first sentence. This is the second sentence.")
for token in doc:
    print(token.i, token.text)

with the output

0 This
1 is
2 the
3 first
4 sentence
5 .
6 This
7 is
8 the
9 second
10 sentence
11 .

How can I get the sentence number as (SENTENCE_NUMBER, token.i, token.text)

0 0 This
0 1 is
0 2 the
0 3 first
0 4 sentence
0 5 .
1 0 This
1 1 is
1 2 the
1 3 second
1 4 sentence
1 5 .

I can reset the token number in the loop, but how can I get the sentence number from doc?

spacy nlp python
2 Answers

There's no built-in sentence index, but you can iterate over sentences:

for sent_i, sent in enumerate(doc.sents):
    for token in sent:
        print(sent_i, token.i, token.text)

You can use custom extensions to save the sentence index on spans or tokens if you need to store it for use elsewhere: https://spacy.io/usage/processing-pipelines#custom-components-attributes

user_4618470

As a list

doc = nlp(u"This is the first sentence. This is the second sentence.")

[sent_id for sent_id, sent in enumerate(doc.sents) for token in sent]

[0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1]

or

doc = nlp(u"This is the first sentence. This is the second sentence.")

[(sent_id, token.i, token.text) for sent_id, sent in enumerate(doc.sents) for token in sent]

[(0, 0, 'This'), (0, 1, 'is'), (0, 2, 'the'), (0, 3, 'first'), (0, 4, 'sentence'), (0, 5, '.'), (1, 6, 'This'), (1, 7, 'is'), (1, 8, 'the'), (1, 9, 'second'), (1, 10, 'sentence'), (1, 11, '.')]

As a numpy array

doc = nlp(u"This is the first sentence. This is the second sentence.")

import numpy as np
np.cumsum(doc.to_array(['SENT_START', ])) - 1

[0 0 0 0 0 0 1 1 1 1 1 1]

As a pandas DataFrame

Using Dframcy from Spacy Universe (pip install dframcy).

doc = nlp(u"This is the first sentence. This is the second sentence.")

from dframcy import DframCy
dframcy = DframCy(nlp)
spacy_df = dframcy.to_dataframe(doc, ['is_sent_start', 'id', 'text', ]).reset_index()
spacy_df.token_is_sent_start = spacy_df.token_is_sent_start.astype(bool).cumsum() - 1
spacy_df = spacy_df.rename(columns={'token_is_sent_start': 'sentence_id',
                                    'index': 'token_id',
                                    'token_text': 'token_text', })

spacy_df

    token_id  sentence_id token_text
0          0            0       This
1          1            0         is
2          2            0        the
3          3            0      first
4          4            0   sentence
5          5            0          .
6          6            1       This
7          7            1         is
8          8            1        the
9          9            1     second
10        10            1   sentence
11        11            1          .
user_62531830
Related