How to merge spacy Vocab instances?

user_1382437 0
Viewed 191

When processing many millions of documents and saving them as spacy documents for later (more processing, visualisation, extracting features), one obvious solution to scaling up is parallel/distributed processing. This means that each parallel process will have their own Vocab that may get extended over time.

When saving documents, clearly one does not want to save the vocab with the document. So to_disk is probably the best option. For loading the document with from_disk, we need a vocab that contains everything from the point in time when the document was processed, or later and from the correct process.

I think it is kind of ugly and clumsy to keep around multiple vocabs and store with each document the id of that vocab. A much better solution would be to create one uber-vocab from the final vocabs from each process after all processes have finished. That vocab could then be used with any document processed.

So the question is: is there already a way to merge a bunch of vocabs into one vocab that would contain the union of all of them?

spacy
1 Answers

Here's a function which creates a new 'merged' vocab instance from a set of vocabs:

import spacy

def merge_vocabs(*vocabs):
    merged = spacy.vocab.Vocab()
    # Inherit some attributes from the first vocab
    # This assumes they are the same across vocabs
    merged.lex_attr_getters = vocabs[0].lex_attr_getters
    merged.morphology = vocabs[0].morphology
    merged.lookups = vocabs[0].lookups
    merged.lookups_extra = vocabs[0].lookups_extra
    # Copy strings and vectors from all vocabs to merged
    for vocab in vocabs:
        for string in vocab.strings:
            string_id = merged.strings.add(string)
            try:
                # Copy vector if exists
                vector = vocab.vectors[string_id]
                merged.set_vector(string_id, vector)
            except KeyError:
                pass
    return merged
            
vocab = merge_vocabs(vocab1, vocab2)

This hasn't been battle-tested but it's worked for me so far.

user_54280130
Related