Spacy: why are vectors not available when using pipe?

user_2098939 0
Viewed 365

This is a very specific question for a very specific case in spacy. I hope Stackoverflow is still the right place to ask this.

There seems to be a difference in the results when using nlp.pipe(docs) vs nlp(doc). When I use the former instead of the latter I don't receive vectors under certain circumstances. Here's what I noticed:

  • When using the default model (en_core_web_sm) with disabled tagger and parser, I get word vectors with nlp(doc) but not with nlp.pipe(docs)
  • Larger models, i.e. en_core_web_lg, generate vectors in both cases
  • Re-enabling the tagger also "fixes" this for the default models
  • Apart from the vectors, disabling the tagger and parser seems to work as expected in both cases

Can somebody explain this to me? Especially why the tagger seems to influence this?

Code:

import spacy


documents = [
    "spaCy is a free, open-source library for advanced Natural Language Processing (NLP) in Python.",
    "If you're working with a lot of text, you'll eventually want to know more about it. For example, what's it about? What do the words mean in context? Who is doing what to whom? What companies and products are mentioned? Which texts are similar to each other?",
    "spaCy is designed specifically for production use and helps you build applications that process and \"understand\" large volumes of text. It can be used to build information extraction or natural language understanding systems, or to pre-process text for deep learning."
]

nlp = spacy.load('en', disable=['tagger', 'parser'])
print('classic model, disabled tagger and parser, without pipe')
for i, doc in enumerate(documents):
    spacy_doc = nlp(doc)
    print(f'document {i} has_vector: {spacy_doc.has_vector}, is_parsed: {spacy_doc.is_parsed}, token_0_pos: {spacy_doc[0].pos_}')

print('classic model, disabled tagger and parser, with pipe')
for i, spacy_doc in enumerate(nlp.pipe(documents)):
    print(f'document {i} has_vector: {spacy_doc.has_vector}, is_parsed: {spacy_doc.is_parsed}, token_0_pos: {spacy_doc[0].pos_}')

nlp = spacy.load('en', disable=['parser'])
print('classic model, disabled only parser, without pipe')
for i, doc in enumerate(documents):
    spacy_doc = nlp(doc)
    print(f'document {i} has_vector: {spacy_doc.has_vector}, is_parsed: {spacy_doc.is_parsed}, token_0_pos: {spacy_doc[0].pos_}')

print('classic model, disabled only parser, with pipe')
for i, spacy_doc in enumerate(nlp.pipe(documents)):
    print(f'document {i} has_vector: {spacy_doc.has_vector}, is_parsed: {spacy_doc.is_parsed}, token_0_pos: {spacy_doc[0].pos_}')

nlp = spacy.load('en', disable=['tagger'])
print('classic model, disabled only tagger, without pipe')
for i, doc in enumerate(documents):
    spacy_doc = nlp(doc)
    print(f'document {i} has_vector: {spacy_doc.has_vector}, is_parsed: {spacy_doc.is_parsed}, token_0_pos: {spacy_doc[0].pos_}')

print('classic model, disabled only tagger, with pipe')
for i, spacy_doc in enumerate(nlp.pipe(documents)):
    print(f'document {i} has_vector: {spacy_doc.has_vector}, is_parsed: {spacy_doc.is_parsed}, token_0_pos: {spacy_doc[0].pos_}')


nlp = spacy.load('en_core_web_lg', disable=['tagger', 'parser'])
print('model en_core_web_lg, disabled tagger and parser, with pipe')
for i, spacy_doc in enumerate(nlp.pipe(documents)):
    print(f'document {i} has_vector: {spacy_doc.has_vector}, is_parsed: {spacy_doc.is_parsed}, token_0_pos: {spacy_doc[0].pos_}')

Output:

classic model, disabled tagger and parser, without pipe
document 0 has_vector: True, is_parsed: False, token_0_pos: 
document 1 has_vector: True, is_parsed: False, token_0_pos: 
document 2 has_vector: True, is_parsed: False, token_0_pos: 
classic model, disabled tagger and parser, with pipe
document 0 has_vector: False, is_parsed: False, token_0_pos: 
document 1 has_vector: False, is_parsed: False, token_0_pos: 
document 2 has_vector: False, is_parsed: False, token_0_pos: 
classic model, disabled only parser, without pipe
document 0 has_vector: True, is_parsed: False, token_0_pos: ADJ
document 1 has_vector: True, is_parsed: False, token_0_pos: ADP
document 2 has_vector: True, is_parsed: False, token_0_pos: ADJ
classic model, disabled only parser, with pipe
document 0 has_vector: True, is_parsed: False, token_0_pos: ADJ
document 1 has_vector: True, is_parsed: False, token_0_pos: ADP
document 2 has_vector: True, is_parsed: False, token_0_pos: ADJ
classic model, disabled only tagger, without pipe
document 0 has_vector: True, is_parsed: True, token_0_pos: 
document 1 has_vector: True, is_parsed: True, token_0_pos: 
document 2 has_vector: True, is_parsed: True, token_0_pos: 
classic model, disabled only tagger, with pipe
document 0 has_vector: False, is_parsed: True, token_0_pos: 
document 1 has_vector: False, is_parsed: True, token_0_pos: 
document 2 has_vector: False, is_parsed: True, token_0_pos: 
model en_core_web_lg, disabled tagger and parser, with pipe
document 0 has_vector: True, is_parsed: False, token_0_pos: 
document 1 has_vector: True, is_parsed: False, token_0_pos: 
document 2 has_vector: True, is_parsed: False, token_0_pos:
spacy
0 Answers
Related