why tf.random.log_uniform_candidate_sampler gives true class?

Viewed 347

I'm reading tensorflow's word2vec tutorial: https://www.tensorflow.org/tutorials/text/word2vec#define_loss_function_and_compile_model

In this tutorial, nagative sampling is conducted using tf.random.log_uniform_candidate_sampler. Given the context class (true class), the goal is to sample negative classes from the whole vocabulary list. To my understanding, The negative classes must differ from the given context class. However, I found that the context class may appear in the negative classes sampled by tf.random.log_uniform_candidate_sampler. Here is the code:

import tensorflow as tf
SEED = 42 

# encode the words
sentence = "The wide road shimmered in the hot sun"
tokens = list(sentence.lower().split())
vocab, index = {}, 1 # start indexing from 1
vocab['<pad>'] = 0 # add a padding token 
for token in tokens:
  if token not in vocab: 
    vocab[token] = index
    index += 1
vocab_size = len(vocab)
print(vocab)
inverse_vocab = {index: token for token, index in vocab.items()}
print(inverse_vocab)


# make (hot, the) as a context pair
target_word, context_word = 6, 1
print("target: {}, context: {}".format(inverse_vocab[target_word], inverse_vocab[context_word]))


# negative sampling
# Set the number of negative samples per positive context. 
num_ns = 4

context_class = tf.reshape(tf.constant(context_word, dtype="int64"), (1, 1))
negative_sampling_candidates, _, _ = tf.random.log_uniform_candidate_sampler(
    true_classes=context_class, # class that should be sampled as 'positive'
    num_true=1, # each positive skip-gram has 1 positive context class
    num_sampled=num_ns, # number of negative context words to sample
    unique=True, # all the negative samples should be unique
    range_max=vocab_size, # pick index of the samples from [0, vocab_size]
    seed=SEED, # seed for reproducibility
    name="negative_sampling" # name of this operation
)
print("negative samples\' index", negative_sampling_candidates)
print("negetive samples: ", [inverse_vocab[index.numpy()] for index in negative_sampling_candidates])
# "the" will show in negative samples, if not, run it several times.

The word the is the context class of word hot, why it could show in the sampled negative classes? Moreover, the target word hot could also be sampled as negative class. Do I misunderstand something?

1 Answers
Related