How to process a very long document in spaCy?

user_16269531 0
Viewed 719

I'm trying to perform a NLP analysis of a text in Spanish. So, to do lemmatization I'm using Spacy, because NLTK has not a Spanish version for lemma. The problem with Spacy is that I a have restriction in the numbers of words that I can pass through Lemmatizer:

ValueError: [E088] Text of length 6095095 exceeds maximum of 1000000. The parser and NER models require roughly 1GB of temporary memory per 100,000 characters in the input. This means long texts may cause memory allocation errors. If you're not using the parser or NER, it's probably safe to increase the nlp.max_length limit. The limit is in number of characters, so you can check whether your inputs are too long by checking len(text).

I tried with nlp.max_length= 6095095 but the session crashed after using all available RAM.

Any suggestion?

spacy lemmatization nlp
1 Answers

You seem to be passing one very long text to spaCy in a single call. Instead of doing that, you can split your text into multiple documents. The exact way to do this depends on your data, but normally you can split something into paragraphs by splitting on double newlines. For example:

import spacy
nlp = spacy.load(... your model ...)

text = ... your text ...
texts = text.split("\n\n") # common way to split on paragraphs

for doc in nlp.pipe(texts):
    ... do something ...
user_3557150
Related