Deal with Out of vocabulary word with Gensim pretrained GloVe

user_11556211 0
Viewed 1345

I am working on an NLP assignment and loaded the GloVe vectors provided by Gensim:

import gensim.downloader
glove_vectors = gensim.downloader.load('glove-twitter-25')

I am trying to get the word embedding for each word in a sentence, but some of them are not in the vocabulary.

What is the best way to deal with it working with the Gensim API?

Thanks!

gensim word-embedding stanford-nlp nlp
1 Answers

Load the model:

import gensim.downloader as api
model = api.load("glove-twitter-25")  # load glove vectors
# model.most_similar("cat")  # show words that similar to word 'cat'

There is a very simple way to find out if the words exist in the model's vocabulary.

result = print('Word exists') if word in model.wv.vocab else print('Word does not exist")

Apart from that, I had used the following logic to create sentence embedding (25 dim) with N tokens:

from __future__ import print_function, division
import os
import re
import sys
import regex
import numpy as np
from functools import partial

from fuzzywuzzy import process
from Levenshtein import ratio as lev_ratio

import gensim
import tempfile


def vocab_check(model, word):
    similar_words = model.most_similar(word)
    match_ratio = 0.
    match_word = ''
    for sim_word, sim_score in similar_words:
        ratio = lev_ratio(word, sim_word)
        if ratio > match_ratio:
            match_word = sim_word
    if match_word == '':
        return similar_words[0][1]
    return model.similarity(word, match_word)


def sentence2vector(model, sent, dim=25):
    words = sent.split(' ')
    emb = [model[w.strip()] for w in words]
    weights = [1. if w in model.wv.vocab else vocab_check(model, w) for w in words]
    
    if len(emb) == 0:
        sent_vec = np.zeros(dim, dtype=np.float16)
    else:
        sent_vec = np.dot(weights, emb)

    sent_vec = sent_vec.astype("float16")
    return sent_vec
user_99395420
Related