Recognize newline (\n) in text as end of sentence in Spacy

user_6599648 0
Viewed 307

I'd like to recognize a newline in text as the end of a sentence. I've tried inputting it into the nlp object like this:

text = 'Guest Blogging\nGuest Blogging allows the user to collect backlinks'
nlp = spacy.load("en_core_web_lg")
config = {"punct_chars": ['\n']}
nlp.add_pipe("sentencizer", config=config)
for sent in nlp(text).sents:
    print('next sentence:')
    print(sent)

The output of this is:

next sentence:
Guest Blogging
Guest Blogging allows the user to collect backlinks

I don't understand why Spacy isn't recognizing the newline as a sentence end. My desired output is:

next sentence:
Guest Blogging:
next sentence:
Guest Blogging allows the user to collect backlinks

Does anyone know how to achieve this?

spacy
2 Answers

The reason the sentencizer isn't doing anything here is that the parser has run first and already set all the sentence boundaries, and then the sentencizer doesn't modify any existing sentence boundaries.

The sentencizer with \n is only the right option if you know you have exactly one sentence per line in your input text. Otherwise a custom component that adds sentence starts after newlines (but doesn't set all sentence boundaries) is probably what you want.

If you want to set some custom sentence boundaries before running the parser, you need be sure you add your custom component before the parser in the pipeline:

nlp.add_pipe("my_component", before="parser")

Your custom component would set token.is_start_start = True for the tokens right after newlines and leave all other tokens unmodified.

Check out the second example here: https://spacy.io/usage/processing-pipelines#custom-components-simple

user_4618470

you can do this by using

    nlp = spacy.load('en_core_web_sm', exclude=["parser"])
    
    text = 'Guest Blogging\nGuest Blogging allows the user to collect backlinks'
    
    config = {"punct_chars": ['\n']}
    nlp.add_pipe("sentencizer", config=config)
    
    for sent in nlp(text).sents:
        print("next sentence")
        print(sent)

Output:

   next sentence
   Guest Blogging
   
   next sentence
   Guest Blogging allows the user to collect backlinks
user_195460870
Related