I am trying to implement variation auto-encoder for real data where both encoder and decoder are modeled via multivariate Gaussian. I have found several implementations online for the case where the encoder is Gaussian and decoder is Bernoulli, but nothing for Gaussian decoder case. For the case of Bernoulli decoder the reconstruction loss can be defined as follows
reconstr_loss = tf.nn.sigmoid_cross_entropy_with_logits(labels=x,logits=x_out_logit)
where x_out_logit is modeled by a DNN. I am not sure how to write reconstruction loss for the Gaussian case. I assumed the decoder should output mean (gz_mean) and variance (gz_log_sigma_sq) as well (similar to the Gaussian encoder) and since the reconstruction loss is the Gaussian probability, I defined it to be
mvn = tf.contrib.distributions.MultivariateNormalDiag(loc=self.gz_mean,scale_diag=tf.sqrt(tf.exp(self.gz_log_sigma_sq)))
reconstr_loss = tf.log(1e-20+mvn.prob(self.x))
However this loss does not seem to work, mvn.prob(self.x) is always zero no matter what training step. Please let me know of any ideas or any git-hub source which considers this case.