Adding documents to gensim model

Viewed 904

I have a class wrapping the various objects required for calculating LSI similarity:

class SimilarityFiles:

    def __init__(self, file_name, tokenized_corpus, stoplist=None):
        if stoplist is None:
            self.filtered_corpus = tokenized_corpus
        else:
            self.filtered_corpus = []
            for convo in tokenized_corpus:
                self.filtered_corpus.append([token for token in convo if token not in stoplist])
        self.dictionary = corpora.Dictionary(self.filtered_corpus)
        self.corpus = [self.dictionary.doc2bow(text) for text in self.filtered_corpus]
        self.lsi = models.LsiModel(self.corpus, id2word=self.dictionary, num_topics=100)
        self.index = similarities.MatrixSimilarity(self.lsi[self.corpus])

I now want to add a function to the class to allow adding documents to the corpus and updating the model accordingly. I've found dictionary.add_documents, and model.add_documents, but there are two things that aren't clear to me:

  1. When you originally create the LSI model, one of the parameters the function receives is id2word=dictionary. When updating the model, how do you tell it to use the updated dictionary? Is it actually unnecessary, or will it make a difference?
  2. How do I update the index? It looks from the documentation that if I use the Similarity class, and not the MatrixSimilarity class, I can add documents to the index, but I don't see such functionality for MatrixSimilarity. If I understood correctly, the MatrixSimilarity is better if my input corpus contains dense vectors (which is does, because I'm using the LSI model). Do I have to change it to Similarity just so that I can update the index? Or, conversely, what's the complexity of creating this index? If it's insignificant, should I just create a new index with my updated corpus, as follows:

Code:

self.dictionary.add_documents(new_docs)    # new_docs is already after filtering stop words
new_corpus = [self.dictionary.doc2bow(text) for text in new_docs]
self.lsi.add_documents(new_corpus)
self.index = similarities.MatrixSimilarity(self.lsi[self.corpus])

Thanks. :)

1 Answers

will it seems that it doesn't update the dictionary.. it just add a new documents not new features.. so you should take a different approach..
I had the same problem and I found this issue on the gensim githup helpful

Related