How to store the Phrase trigrams gensim model after training

user_2160884 0
Viewed 56

I would like to know can I store the gensim Phrase model after training it on the sentences

documents = ["the mayor of new york was there", "human computer interaction and 
machine learning has now become a trending research area","human computer interaction 
is interesting","human computer interaction is a pretty interesting subject", "human 
computer interaction is a great and new subject", "machine learning can be useful 
sometimes","new york mayor was present", "I love machine learning because it is a new 
subject area", "human computer interaction helps people to get user friendly 
applications"]

sentences = [doc.split(" ") for doc in documents]

bigram_transformer = Phrases(sentences)
bigram_sentences = bigram_transformer[sentences]
print("Bigrams - done")
# Here we use a phrase model that detects the collocation of 3 words (trigrams).
trigram_transformer = Phrases(bigram_sentences)
trigram_sentences = trigram_transformer[bigram_sentences]
print("Trigrams - done")

How to store trigram_transformer physically to reuse it again using pickle maybe?

Thank you in advance for your help.

gensim nlp python
2 Answers

Convert list or that partular format into an numpy array and save it as a .npy file easy to save and easy to read, using this by numpy gives you advantage of loading it in almost every platform like google colab, replit ..... refer to this link for more details on saving a npy file numpy.save()

Using pickle is also a good option but things get a bit tricky at points when difference in encoding standards and such problems arise.

user_124432600

You can use Gensim's native .save() method:

trigram_transformer.save(TRIPHRASER_PATH)

...then reload similarly:

reloads_trigram_transformer = Phrases.load(TRIPHRASER_PATH)

(The Gensim save/load methods generally use Python pickling, but may for some models & version-transitions handle some properties specially.)

You could also use Python's own pickle, which should work OK unless/until you try to load a too-old model into a newer version of Gensim that might have changed things about the Phrases model.

user_1302880
Related