I'm implementing the reconstruction probability of VAE in paper "Variational Autoencoder based Anomaly Detection using Reconstruction Probability". But I got a problem with the shape of mean_x' and sigma_x' for multivariate normal distribution.
The logic of reconstruction probability is:
- input x to Encoder, get the mean(mu_z) and variance(sigma_z) of latent vector;
- take L sample from the mu_z and sigma_z;
- input these L samples into Decoder, get the mean(mu_x') and variance(sigma_x') of generated x';
- calculate the log probability of x in the multivariate normal distribution with mu_z' and sigma_x';
As I know, the shape of mu_x' and sigma_x' is the same when output from the decoder and should be (sequence_length,batch_size,num_of_features), like (240,32,1). To calculate the multivariate normal distribution log_probability, we should input these two parameters into distribution, but as the distribution parameter, the covariance matrix should not be as the same shape as the mean. So how should I convert sigma_x' and input it into MultivariateNormal function? Here's my code. The "#Problem line" is the line with the problem.
reconstructed_prob = np.zeros((len(dataset),), dtype='float32')
test_loader = DataLoader(dataset=dataset,
batch_size=self.batch_size,
shuffle=False,
drop_last=True)
if self.is_fitted:
with torch.no_grad():
for t, x in enumerate(test_loader): #t=0, x.shape(32,240,1)
x = x[0]
x = x.permute(1, 0, 2) #(240,32,1)
x = Variable(x[:, :, :].type(self.dtype), requires_grad=True)
_, _, cell_output = self.encoder(x) # h_hidden, (batch_size,units) (32,90)
latent_mean, latent_logvar = self.lmbd(cell_output) # input: last hidden state of encoder return: latent vector(batch_size32, z_dims5)
score = 0
for _ in range(L):
# Sample a random c and z vector and reparametrize
# epsilon_c = torch.randn(net.batch_size, net.T, net.z_dim)
epsilon_z = torch.randn_like(latent_mean)
if torch.cuda.is_available():
# epsilon_c = epsilon_c.to(torch.device(0))
epsilon_z = epsilon_z.to(torch.device(0))
# c = mu_c + epsilon_c * sigma_c
z = epsilon_z.mul(latent_logvar).add_(latent_mean) #(32,5)
# Pass sample through decoder and calculate reconstruction prob
mu_x, sigma_x = self.decoder(z) # (240,32,1)
pdf = torch.distributions.multivariate_normal.MultivariateNormal(mu_x,sigma_x) #problem line
score += pdf.log_prob(x)
p_l = multivariate_normal.pdf(x, mu_x, sigma_x)
reconstructed_prob[t] = score/L