that might actually be a dumb question but I just can't figure out why my script with gensim.models.word2vec is not working. Here is the thing, I'm using the stanford sentiment analysis databank dataset (~11000 reviews), and i'm trying to build word2vec using gensim, this is my script:
import gensim as gs
import sys
# open the datas
sentences = gs.models.word2vec.LineSentence('../processedWords.txt')
print("size in RAM of the sentences: {}".format(sys.getsizeof(sentences)))
# transform them
# bigram_transformer = gs.models.Phrases(sentences)
model = gs.models.word2vec.Word2Vec(sentences, min_count=10, size=100, window=5)
model.save('firstModel')
print(model.similarity('film', 'test'))
print(model.similarity('film', 'movie'))
Now, my problem is that the script runs in 2s, and gives only huge similarity between every pair of words. In addition, some words which are in the sentences are not in the built vocabulary.
I must be doing something obviously wrong, but can't figure what.
Thank you for your help.