Convert PySpark ML Word2Vec model to Gensim Word2Vec model

user_9898446 0
Viewed 1075

I've generated a PySpark Word2Vec model like so:

from pyspark.ml.feature import Word2Vec

w2v = Word2Vec(vectorSize=100, minCount=1, inputCol='words', outputCol = 'vector')
model = w2v.fit(df)

(The data that I used to train the model on isn't relevant, what's important is that its all in the right format and successfully yields a pyspark.ml.feature.Word2VecModel object.)

Now I need to convert this model to a Gensim Word2Vec model. How would I go about this?

word2vec gensim pyspark
1 Answers

If you still have the training data, re-training the gensim Word2Vec model may be the most straightforward approach.

If you only need the word-vectors, perhaps PySpark's model can export them in the word2vec.c format that gensim can load with .load_word2vec_format().

The only reason to port the model would be to continue training. Such incremental training, while possible, involves considering a lot of tradeoffs in balancing the influence of the older and later training to get good results.

If you are in fact wanting to do this conversion in order to do more training in such a manner, it again suggests that using the original training to reproduce a similar model could be plausible.

But, if you have to convert the model, the general approach would be to study the source code and internal data structures of the two models, to discover how they alternatively represent each of the key aspects of the model:

  • the known word-vectors (model.wv.vectors in gensim)
  • the known-vocabulary of words, including stats about word-frequencies and the position of individual words (model.wv.vocab in gensim)
  • the hidden-to-output weights of the model (`model.trainables' and its properties in gensim)
  • other model properties describing the model's modes & metaparameters

A reasonable interactive approach could be:

  • Write some acceptance tests that take models of both types, and test whether they are truly 'equivalent' for your purposes. (This is relatively easy for just checking if the vectors for individual words are present and identical, but nearly as hard as the conversion itself for verifying other ready-to-be-trained-more behaviors.)

  • Then, in an interactive notebook, load the source model, and also create a dummy gensim model with the same vocabulary size. Consulting the source code, write Python statements to iteratively copy/transform key properties over from the source into the target, repeatedly testing if they verify as equivalent.

  • When they do, take those steps you did manually and combine them into a utility method to do the conversion. Again verify its operation then try using the converted model however you'd hoped – perhaps discovering overlooked info or discovering other bugs in the process, and then improving the verification method and conversion method.

It's possible that the PySpark model will be missing things the gensim model expects, which might require synthesizing workable replacement values.

Good luck! (But re-train the gensim model from the original data if you want things to just be straightforward and work.)

user_1302880
Related