Spacy adds words automatically to vocab?

user_18319078 0
Viewed 303

I loaded regular spacy language, and tries the following code:

import spacy

nlp = spacy.load("en_core_web_md")

text = "xxasdfdsfsdzz is the first U.S. public company"

if 'xxasdfdsfsdzz' in nlp.vocab:
    print("in")
else:
    print("not")
    
if 'Apple' in nlp.vocab:
    print("in")
else:
    print("not")


# Process the text
doc = nlp(text)

if 'xxasdfdsfsdzz' in nlp.vocab:
    print("in")
else:
    print("not")
    
if 'Apple' in nlp.vocab:
    print("in")
else:
    print("not")

It seems like spacy loaded words after they called to analyze - nlp(text) Can someone explain the output? How can I avoid it? Why "Apple" is not existing in vocab? and why "xxasdfdsfsdzz" exists?

Output:

not
not
in
not
spacy nlp python-3.x python
1 Answers

The spaCy Vocab is mainly an internal implementation detail to interface with a memory-efficient method of storing strings. It is definitely not a list of "real words" or any other thing that you are likely to find useful.

The main thing a Vocab stores by default is strings that are used internally, such as POS and dependency labels. In pipelines with vectors, words in the vectors are also included. You can read more about the implementation details here.

All words an nlp object has seen need storage for their strings, and so will be present in the Vocab. That's what you're seeing with your nonsense string in the example above.

user_3557150
Related