My environment
- MacOS 10.15 / Debian 11
- Python 3.8.2 / 3.8.12
- Spacy 3.1.2
- Spacy-langdetect 0.1.2
I'm trying to use spacy-langdetect to add a language detection feature in my spaCy NLP pipeline. Everything looks good when I use a single process to perform the detection like in the following example
import spacy
from spacy_langdetect import LanguageDetector
from spacy.language import Language
# Load Language detection
@Language.factory('language_detector')
def language_detector(nlp, name):
return LanguageDetector()
# Load Spacy
nlp = spacy.load("en_core_web_lg")
nlp.add_pipe('language_detector', last=True)
print([doc._.language for doc in nlp.pipe(['I bless the rains down in Africa'], n_process=-1)])
returns
[{'language': 'en', 'score': 0.9999985260933938}]
but I got the following error when I set n_process > 1 :
AttributeError: [E046] Can't retrieve unregistered extension attribute 'language'. Did you forget to call the `set_extension` method?
I thought it was due to the usage of spawn method (in multiprocessing) on macOS since Python 3.8 and some context not sent to the sub-process but I got the same error on Linux using fork method. Has anyone an explanation for this error and any workaround?