I'm trying to implement seq2seq with a feature to generate sampled outputs from the decoder i.e. at every step, rather than taking the argmax of the output logits from previous state, it should sample from them according to the logit distribution and use that as input for the next step.
After poking around I found the loop_function in seq2seq.py as a promising place to start. It looks like I have to write a loop function that looks like this (modified from the one in the file that extracts argmax+embedding):
def _extract_sample_and_embed(embedding, output_projection=None,
update_embedding=True):
def loop_function(prev, _):
if output_projection is not None:
prev = nn_ops.xw_plus_b(prev, output_projection[0], output_projection[1])
prev_symbol = math_ops.sample(prev) #<------- Need this op but it does not exist ?
emb_prev = embedding_ops.embedding_lookup(embedding, prev_symbol)
if not update_embedding:
emb_prev = array_ops.stop_gradient(emb_prev)
return emb_prev
return loop_function
Then i use this loop function generator in the seq2seq_embedding_with_attention model. However, the op I need that samples from a tensor of floats does not exist in tensorflow, so do I need to write my own? How do I do that?
In searching for guidance, I found that in tensorflow/tensorflow/python/ops/candidate_sampling_ops there is a reference to :
from tensorflow.python.ops import gen_candidate_sampling_opsbut I can't find this file. I'm guessing it's auto-generated from somewhere. where?