How to update .mm (market matrix) file with new docs (corpus)?

user_5741062 0
Viewed 130

I am looking for a way to update the existing corpus with new docs using gensim. Here, I have created a dictionary from the existing corpus and a bag of words for the same. Later, I serialized it to .mm file and saved it to the disk locally. Now, I want to update my existing .mm file with new docs so that I can keep the representation of updated corpus so that on unseen data I will be able to use it for document similarity. Please assist me that how can I do it? What is the proper way to update the corpus? Moreover, I am aware that instead of .mm file I can add documents to the dictionary.

from gensim import corpora, models, similarities
from gensim.parsing.preprocessing import STOPWORDS

tweets = [
    ['human', 'interface', 'computer'],
    ['survey', 'user', 'computer', 'system', 'response', 'time', 'survey'],
    ['eps', 'user', 'interface', 'system'],
    ['system', 'human', 'system', 'eps'],
    ['user', 'response', 'time'],
    ['trees'],
    ['graph', 'trees'],
    ['graph', 'minors', 'trees'],
    ['graph', 'minors', 'survey']
]

dictionary = corpora.Dictionary(tweets)
dictionary.save('tweets.dict')  # store the dictionary, for future reference

dictionary = corpora.Dictionary.load('tweets.dict')
print(f'Length of previous dict = {len(dictionary)}, tokens = {dictionary.token2id}')
raw_corpus = [dictionary.doc2bow(t) for t in tweets]
corpora.MmCorpus.serialize('tweets.mm', raw_corpus)  # store to disk
print("Save the vectorized corpus as a .mm file")

corpus = corpora.MmCorpus('tweets.mm') # loading saved .mm file
print(corpus)

new_docs = [
["user", "response", "system"],
["trees", "minor", "surveys"]
]

# how to add this new_docs corpus to tweets.mm

Can tweets.mm be updated? Or is it recommended?

word2vec gensim similarity nlp python
1 Answers

There is no direct way to update .mm corpus on disk. Instead, I suggest that you read in the corpus from the file and re-process it from scratch by extending the tweets list with the contents of new_docs. In such a way you ensure that the dictionary (mapping of words to ids) in your corpus is not going to get desynchronized with the corpus.

I would go with creating the following function that would deal with the updates:

def update_corpus(tweets, new_docs, dict_path):
    dictionary = corpora.Dictionary.load(dict_path)
    print(f'Length of previous dict = {len(dictionary)}, tokens = {dictionary.token2id}')
    dictionary.add_documents(new_docs)
    dictionary.save(dict_path)
    print(f'Length of updated dict = {len(dictionary)}, tokens = {dictionary.token2id}')
    import itertools  # you can move it outside of the function
    full_corpus = itertools.chain(tweets, new_docs)
    raw_corpus = [dictionary.doc2bow(t) for t in full_corpus]
    corpora.MmCorpus.serialize('tweets.mm', raw_corpus)  # store to disk
    print("Save the vectorized corpus as a .mm file")

Please note that there is no need to load the dictionary just after it was created and saved, so you can remove this line:

dictionary = corpora.Dictionary.load('tweets.dict')
user_65739020
Related