I'm trying to reduce the dimensionality of a TFIDF vector using LSI. When I pass the TFIDF vector to the LSI model, I get a segmentation fault. No idea why. Is it not enough memory on my machine (16gb RAM Macbook) or perhaps something strange with a library install?
import gensim.models.lsimodel
from gensim.corpora import Dictionary
import logging
logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.DEBUG, handlers=[logging.StreamHandler()])
dictionary = Dictionary.load(GENSIM_DICT_PATH + '_testfullsize') # created while making the TFIDF models
lsimodel = gensim.models.LsiModel(id2word=dictionary, num_topics=200)
records = load_zipped_pickle(file)
formatted_records = [record['vecs']['tfidf'] for record in records]
chunksize = 1
lsimodel.add_documents(formatted_records[:2], chunksize=chunksize)
2022-09-11 18:30:02,787 : INFO : loading Dictionary object from /dev/data/poc/preprocess/gensim_dict/gensim_dict_testfullsize
2022-09-11 18:30:02,787 : DEBUG : {'uri': /dev/data/poc/preprocess/gensim_dict/gensim_dict_testfullsize', 'mode': 'rb', 'buffering': -1, 'encoding': None, 'errors': None, 'newline': None, 'closefd': True, 'opener': None, 'ignore_ext': False, 'compression': None, 'transport_params': None}
2022-09-11 18:30:02,830 : INFO : Dictionary lifecycle event {'fname': /dev/data/poc/preprocess/gensim_dict/gensim_dict_testfullsize', 'datetime': '2022-09-11T18:30:02.797739', 'gensim': '4.2.0', 'python': '3.9.12 (main, Apr 5 2022, 01:53:17) \n[Clang 12.0.0 ]', 'platform': 'macOS-10.16-x86_64-i386-64bit', 'event': 'loaded'}
2022-09-11 18:30:02,832 : INFO : using serial LSI version on this node
2022-09-11 18:30:02,916 : INFO : updating model with new documents
2022-09-11 18:30:02,916 : INFO : preparing a new chunk of documents
2022-09-11 18:30:02,916 : DEBUG : converting corpus to csc format
2022-09-11 18:30:02,917 : INFO : using 100 extra samples and 2 power iterations
2022-09-11 18:30:02,917 : INFO : 1st phase: constructing (30719, 2100) action matrix
2022-09-11 18:30:02,922 : INFO : orthonormalizing (30719, 2100) action matrix
2022-09-11 18:30:03,342 : DEBUG : computing QR of (30719, 2100) dense matrix
2022-09-11 18:30:46,288 : DEBUG : running 2 power iterations
2022-09-11 18:30:47,006 : DEBUG : computing QR of (30719, 2100) dense matrix
2022-09-11 18:31:17,479 : DEBUG : computing QR of (30719, 2100) dense matrix
2022-09-11 18:31:43,074 : INFO : 2nd phase: running dense svd on (2100, 1) matrix
2022-09-11 18:31:43,296 : INFO : computing the final decomposition
2022-09-11 18:31:43,299 : INFO : keeping 1 factors (discarding 0.000% of energy spectrum)
2022-09-11 18:31:43,331 : INFO : processed documents up to #1
2022-09-11 18:31:43,347 : INFO : topic #0(1.000): 0.848*"andrews" + 0.216*"appellate" + 0.182*"safety" + 0.181*"vii" + 0.170*"unclaimed" + 0.139*"christina" + 0.136*"serie" + 0.108*"terminable" + 0.103*"impedimento" + 0.091*"contemporaneously"
2022-09-11 18:31:43,347 : INFO : preparing a new chunk of documents
2022-09-11 18:31:43,347 : DEBUG : converting corpus to csc format
2022-09-11 18:31:43,349 : INFO : using 100 extra samples and 2 power iterations
2022-09-11 18:31:43,350 : INFO : 1st phase: constructing (30719, 2100) action matrix
zsh: segmentation fault ipython
Stats on the contents of records:
len(records)
100 records in the file
formatted_records[0][:5]
[(0, 0.008193880705989672),
(1, 0.002169626928118455),
(2, 0.001453701764945406),
(5, 0.004775151397123477),
(6, 0.00426000261783023)]
numpy.mean([len(x) for x in formatted_records])
1036.82
numpy.median([len(x) for x in formatted_records])
830.5
Incidentally, I'm using Gensim for both creating the TFIDF vector and LSI, so if there's a better way to do this entire operation, I'd appreciate pointers.
tfidfmodel = gensim.models.tfidfmodel.TfidfModel(dictionary=gensim_dict)
record['vecs']['tfidf'] is produced using tfidfmodel[gensim_dict.doc2bow(text)]