I am using gensim's lda implementation to create a document topic model distribution. I have a model lda trained
dictionary = corpora.Dictionary(data)
corpus = [dictionary.doc2bow(doc) for doc in data]
num_cores = multiprocessing.cpu_count()
num_topics = 150
lda = LdaMulticore(corpus=corpus, num_topics=num_topics, id2word=dictionary, workers=num_cores, alpha=1e-4, eta=5e-1,
minimum_probability=0.0)
I want to extract a distribution matrix of size N * M where N is the number of documents in the corpus, and M is the number of topics. Each row therefore would represent a document, and each column would represent a topic. These distributions are already available in the lda model variable e.g.
lda
>>> [[(0,0.01),(1,0.23),(2,1e-7)],
[(0,0.91),(1,0.067),(2,0.38)]]
where the above represents a simple example if we had 2 documents and 3 topics (0,1,2). So lda is a list of list of tuples, where each list is a document, the first element in the tuple is the topic id and the second element is the topic contribution to that document.
What I have done so far
# approach 1
distributions = np.array( [[tup[1] for tup in lst] for lst in lda[corpus]] )
# approach 2 (about 3X slower than approach 1)
distributions = np.array(lda[corpus])[:,:,1]
The problem is, for a dataset with 300k documents and 150 topics, creating the distributions matrix using approach 1 takes over 10 minutes.
Ultimately, I want to create this matrix to feed into a function which calculates document similarity (comparing the first row / document of the matrix to all other rows / documents). If there is a way to directly feed the learned lda model values into the function, that would be great as it would save time on creating the redundant matrix. If not, is there a more efficient way to create the distributions matrix?
For reference, the document comparison function into which the distributions matrix is fed is
from sklearn.preprocessing import normalize
from scipy.stats import entropy
def jsd(mat):
mat = normalize(mat, axis=1, norm='l1')
p = mat[0,None].T # just comparing first row to all other rows
q = mat[0:].T
m = 0.5*(p + q)
return 0.5*(entropy(p,m) + entropy(q,m))