I'm reading tensorflow's word2vec tutorial: https://www.tensorflow.org/tutorials/text/word2vec#define_loss_function_and_compile_model
In this tutorial, nagative sampling is conducted using tf.random.log_uniform_candidate_sampler. Given the context class (true class), the goal is to sample negative classes from the whole vocabulary list. To my understanding, The negative classes must differ from the given context class. However, I found that the context class may appear in the negative classes sampled by tf.random.log_uniform_candidate_sampler. Here is the code:
import tensorflow as tf
SEED = 42
# encode the words
sentence = "The wide road shimmered in the hot sun"
tokens = list(sentence.lower().split())
vocab, index = {}, 1 # start indexing from 1
vocab['<pad>'] = 0 # add a padding token
for token in tokens:
if token not in vocab:
vocab[token] = index
index += 1
vocab_size = len(vocab)
print(vocab)
inverse_vocab = {index: token for token, index in vocab.items()}
print(inverse_vocab)
# make (hot, the) as a context pair
target_word, context_word = 6, 1
print("target: {}, context: {}".format(inverse_vocab[target_word], inverse_vocab[context_word]))
# negative sampling
# Set the number of negative samples per positive context.
num_ns = 4
context_class = tf.reshape(tf.constant(context_word, dtype="int64"), (1, 1))
negative_sampling_candidates, _, _ = tf.random.log_uniform_candidate_sampler(
true_classes=context_class, # class that should be sampled as 'positive'
num_true=1, # each positive skip-gram has 1 positive context class
num_sampled=num_ns, # number of negative context words to sample
unique=True, # all the negative samples should be unique
range_max=vocab_size, # pick index of the samples from [0, vocab_size]
seed=SEED, # seed for reproducibility
name="negative_sampling" # name of this operation
)
print("negative samples\' index", negative_sampling_candidates)
print("negetive samples: ", [inverse_vocab[index.numpy()] for index in negative_sampling_candidates])
# "the" will show in negative samples, if not, run it several times.
The word the is the context class of word hot, why it could show in the sampled negative classes? Moreover, the target word hot could also be sampled as negative class. Do I misunderstand something?