spacy - removing a token from doc while preserving span attributes

user_13331524 0
Viewed 37

I would like to add information about paragraphes and headings on my spacy document. For this I have added beacon in between the paragraphs text and heading (such as < p_start > for the beginning of a paragraph).

For this I have placed the custom functions that detect these beacon and tagged the spans as paragraphs and/or headings, after the tokenizer, but before the tok2vec component. Thus the pipeline tokenize the text, the tag the spans, then apply the regular pipeline components.

Now I have a problem as I don't want these beacons to be processed in my final doc. However I couldn't find a way, either to remove such token during the pipeline processing, or even substitute them into whitespace.

So is there a way to change these tagged documents, in order to remove only the non relevant tokens ?

spacy python
1 Answers

There is no way to change the text of a spaCy Doc after it's created. That's a design decision to avoid data loss.

If you need to modify the contents of a Doc after creating it, what you can do is create a new Doc and pass it to a different pipeline. If you pass a Doc rather than a string as input to nlp then tokenization will be skipped.

user_3557150
Related