How keras oov_token=True works

user_3481069 0
Viewed 331

By default how does keras impute out-of-vocabulary token when oov_token=True.

According to keras official documentation it tells that if given, it will be added to word_index and used to replace out-of-vocabulary words during text_to_sequence calls. However, no much details when not explicitly specified but oov_token=True.

recurrent-neural-network keras tensorflow nlp python
1 Answers

Assuming, you are referring to the oov_token of the tf.keras.preprocessing.text.Tokenizer, you should take a look at the source code to understand what is happening under the hood. In the text_to_sequence method, you see that the index of the oov_token is added on two occasions for oov_token=True:

  1. When a word in a sequence is not found in word_index, which is the dictionary of each word in your vocabulary mapped to a unique integer value.
  2. When you set the maximum number of words to keep using num_words and i being the index of a certain word, is equals to or above num_words.

Here is the related code:

vect = []
for w in seq:
    i = self.word_index.get(w)
    if i is not None:
        if num_words and i >= num_words:
            if oov_token_index is not None:
                vect.append(oov_token_index)
        else:
            vect.append(i)
    elif self.oov_token is not None:
        vect.append(oov_token_index)
yield vect

Also, here you see that the oov_token always gets the index 1 if it was set to True.

user_96578610
Related