Facebook fasttext bin model preprocessing

Viewed 28

I have downloaded a .bin FastText model,and load it as follows:

ft = fasttext.load_model('/content/drive/MyDrive/dataset/cc.en.300.bin')

how can i make preprocessing and normalization on cc.en.300.bin model. i want to make lemmatization, removing stopwords and other operation

1 Answers

Your question doesn't really make sense, given how FastText models are usually used, on a couple levels:

  • A pre-trained FastText model, like cc.en.300.bin, no longer has any original text inside it, as would normally be the input for preprocessing/normalization. It is the end result of someone else's training on a corpus they already prepared for the FastText model. Essentially, you're stuck with their choices of tokenization, normalization, & other preprocessing.

  • Because FastText models learn from the same kind of word-morphology (including roots/stems/alternate-forms) that's removed by stemming/lemmatization, generally the training texts used for FastText training aren't preprocessed in that way. And, you wouldn't perform any such transformation on word-tokens you want to look-up in such a pre-trained model.

The only way I can imagine your question representing a real need is if you already have some other texts that have been destructively preprocessed – such as by replacing words with their stems/lemmas – and now you want to look up those words in this model.

If that's your true need:

  1. You should try to go back to texts that weren't destructively changed, and look up the original (non-lemmatized) words in the FastText model. That's the usual way to use FastText, and what I'd expect to work best. (For example, looking up each of 'walking', 'walked', 'walks', etc will ikely give better vectors than reducing them all to the lemma 'walk' and only looking that up.) If you can't recover the original words...

  2. You could try just looking up the lemmas/etc you have directly. The FastText model will give you its vector for that root word, synthesizing a guess-vector if necessary from the word-fragments it knows. That might work fine. Or...

  3. You could conceivably iterate over all the model's known words, and map them to the alernate/normalized words of your preprocessing scheme. This would potentially map N of known words to 1 new 'reduced' word. Then, create a new model where that single reduced word, in your new scheme, has just one vector – perhaps by some sort of (weighted) average of all the others words it might have been, before the reductive canonicalization. But, I have a hard time imagining any situation where this extra complexity would offer any advantage over either (1) above, using FastText as intended by looking up the words that are already in the model, or (2) above, just settling for what it already has for your new word. So really, don't do anything like this unless you've got some good reason.

Related