Discrepancy between tensorflow's conv1d and pytorch's conv1d

user_3086871 0
Viewed 862

I am trying to import some pytorch code to tensorflow, I came to know that torch.nn.functional.conv1d() is tf.nn.conv1d() but I am afraid there are still some discrepancies in tf's versions. Specifically, I cannot find the group parameter in tf.conv1d. For example: the following codes output two different results:

Pytorch: 

inputs = torch.Tensor([[[1, 1, 1, 1],[2, 2, 2, 2],[3, 3, 3, 3]]]) #batch_sizex seq_length x embed_dim, 
inputs = inputs.transpose(2,1) #batch_size x embed_dim x seq_length
batch_size, embed_dim, seq_length = inputs.size()

kernel_size = 3
in_channels = 2
out_channels = in_channels 
weight = torch.ones(out_channels, 1, kernel_size)

inputs = inputs.contiguous().view(-1, in_channels, seq_length) #batch_size*embed_dim/in_channels x in_channels x seq_length 
inputs = F.pad(inputs, (kernel_size-1,0), 'constant', 0)
output = F.conv1d(inputs, weight, padding=0, groups=in_channels)
output = output.contiguous().view(batch_size, embed_dim, seq_length).transpose(2,1)

Output:

tensor([[[1., 1., 1., 1.],
         [3., 3., 3., 3.],
         [6., 6., 6., 6.]]])

Tensorflow: 

inputs = tf.constant([[[1, 1, 1, 1],[2, 2, 2, 2],[3, 3, 3, 3]]], dtype=tf.float32) #batch_sizex seq_length x embed_dim
inputs = tf.transpose(inputs, perm=[0,2,1])
batch_size, embed_dim, seq_length = inputs.get_shape()

print(batch_size, seq_length, embed_dim)
kernel_size = 3
in_channels = 2
out_channels = in_channels 
weight = tf.ones([kernel_size, in_channels, out_channels])

inputs = tf.reshape(inputs, [(batch_size*embed_dim)//in_channels, in_channels, seq_length], name='inputs')
inputs = tf.transpose(inputs, perm=[0, 2, 1])
padding = [[0, 0], [(kernel_size - 1), 0], [0, 0]]
padded = tf.pad(inputs, padding)

res = tf.nn.conv1d(padded, weight, 1, 'VALID')
res = tf.transpose(res, perm=[0, 2, 1])
res = tf.reshape(res, [batch_size, embed_dim, seq_length])
res = tf.transpose(res, perm=[0, 2, 1])
print(res)

Output: 

[[[ 2.  2.  2.  2.]
  [ 6.  6.  6.  6.]
  [12. 12. 12. 12.]]], shape=(1, 3, 4), dtype=float32)
pytorch tensorflow convolution
1 Answers

Different results

There is no discrepancy between those versions, you are just setting up different things. To get exactly same results as in Tensorflow change the lines specifying weights to:

weight = torch.ones(out_channels, 2, kernel_size)

, because your input has two input channels, as you have correctly declared in TF:

weight = tf.ones([kernel_size, in_channels, out_channels])

Groups parameter

You have misunderstood what is groups parameter responsible for in pytorch. It restricts the number of channels each filter uses (in this case only one as 2 input_channels divided by 2 give us one).

See here for more intuitive explanation for 2D convolution.

user_108864200
Related