How to average the vector when merging (with retokenize) custom noun chunks in spaCy?

user_12584467 0
Viewed 170

I am generating noun chunks using spaCy's statistical models (e.g. noun chunks based on the part-of-speech tags and dependencies) and rule-based matching to capture (user supplied) context specific noun chunks.

For other downstream tasks, I am retokenizing these noun chunks (spans), which works fine for the most part. However, the token's vector representation (token.vector) gets set to all zeros. Is there a way to retain the vector information e.g. by averaging the individual token vectors and assigning it to the retokenised token?

I tried this with the code...

def tokenise_noun_chunks(doc)
    if not doc.has_annotation("DEP"):
        return doc

    all_noun_chunks = list(doc.noun_chunks) + doc._.custom_noun_chunks

    with doc.retokenize() as retokenizer:
        for span in all_noun_chunks:
            # if I print(span.vector) here, I get the correctly averaged vector
            attrs = {"tag": span.root.tag, "dep": span.root.dep}
            retokenizer.merge(np, attrs=attrs)
        return doc

...but when I check the returned vectors for the noun chunks, I get a zeros array. I've modelled this (the above code) on the built-in merge_noun_chunks component (just modified to include my own custom noun chunks), so I can confirm that the built in component gives the same results.

Is there any way to keep the word vector information? Will I need to add the term/average vector to the Vector store?

spacy nlp python
1 Answers

The retokenizer should set span.vector as the vector for the new merged token. With spacy==3.0.3 and en_core_web_md==3.0.0:

import spacy
nlp = spacy.load("en_core_web_md")
doc = nlp("This is a sentence.")
with doc.retokenize() as retokenizer:
    for chunk in doc.noun_chunks:
        retokenizer.merge(chunk)
for token in doc:
    print(token, token.vector[:5])

Output:

This [-0.087595  0.35502   0.063868  0.29292  -0.23635 ]
is [-0.084961   0.502      0.0023823 -0.16755    0.30721  ]
a sentence [-0.093156   0.1371495 -0.307255   0.2993     0.1383735]
. [ 0.012001  0.20751  -0.12578  -0.59325   0.12525 ]

Attributes like tag and dep are also set to those of span.root by default, so you only need to specify them if you want to override the defaults.

user_4618470
Related