I have a class wrapping the various objects required for calculating LSI similarity:
class SimilarityFiles:
def __init__(self, file_name, tokenized_corpus, stoplist=None):
if stoplist is None:
self.filtered_corpus = tokenized_corpus
else:
self.filtered_corpus = []
for convo in tokenized_corpus:
self.filtered_corpus.append([token for token in convo if token not in stoplist])
self.dictionary = corpora.Dictionary(self.filtered_corpus)
self.corpus = [self.dictionary.doc2bow(text) for text in self.filtered_corpus]
self.lsi = models.LsiModel(self.corpus, id2word=self.dictionary, num_topics=100)
self.index = similarities.MatrixSimilarity(self.lsi[self.corpus])
I now want to add a function to the class to allow adding documents to the corpus and updating the model accordingly.
I've found dictionary.add_documents, and model.add_documents, but there are two things that aren't clear to me:
- When you originally create the LSI model, one of the parameters the function receives is
id2word=dictionary. When updating the model, how do you tell it to use the updated dictionary? Is it actually unnecessary, or will it make a difference? - How do I update the index? It looks from the documentation that if I use the
Similarityclass, and not theMatrixSimilarityclass, I can add documents to the index, but I don't see such functionality forMatrixSimilarity. If I understood correctly, theMatrixSimilarityis better if my input corpus contains dense vectors (which is does, because I'm using the LSI model). Do I have to change it toSimilarityjust so that I can update the index? Or, conversely, what's the complexity of creating this index? If it's insignificant, should I just create a new index with my updated corpus, as follows:
Code:
self.dictionary.add_documents(new_docs) # new_docs is already after filtering stop words
new_corpus = [self.dictionary.doc2bow(text) for text in new_docs]
self.lsi.add_documents(new_corpus)
self.index = similarities.MatrixSimilarity(self.lsi[self.corpus])
Thanks. :)