Pytorch: how to mask flexible size of input for average pooling?

user_3293726 0
Viewed 1628

Backgroud: I would like to apply convolutions and average pooling to sequences for sequence classification task.

Problem: without mask for inputs, the calculation of mini-batch and one-by-one will be different. For example:

# Two sequences.
s1 = torch.range(start=1, end=6).view(-1, 1)
s2 = torch.range(start=1, end=3).view(-1, 1)

##########################################
# one-by-one

# Convolutions.
kernels = torch.ones(1, 1, 2)
h1 = F.conv1d(s1.view(1, 1, -1), kernels) #h1=[[[3, 5, 7, 9, 11]]]
h2 = F.conv1d(s2.view(1, 1, -1), kernels) #h2=[[[3, 5]]]

# Average pooling.
h1 = h1.mean(-1) #h1=[[7]]
h2 = h2.mean(-1) #h2=[[4]]

##########################################
# mini-batch
s = torch.nn.utils.rnn.pad_sequence([s1, s2], batch_first=True)
s = s.permute(0, 2, 1)
h = torch.mean(F.conv1d(s, kernels), dim=-1) #h=[[7], [2.2]]

As you can see, the h is different from [h1, h2].

Questions:

  1. In order to make mini-batch the same as one-by-one, what is the normal way to employ?
  2. Is there a given pytorch api?
  3. If not, how to implement that?
conv-neural-network pytorch deep-learning time-series nlp
1 Answers

Ideally, this will not be an issue any more when pytorch convolutions and pooling operations will support packed sequence, but this is not yet the case afaik.

So, you have to deal with this yourself. There are 2 issues to deal with:

  • at the right border, convolutions with padding may give more outputs than without padding
  • your mean() operation will take into account padding

One possible solution would be to compute a mask tensor that masks with 0 all convolution outputs that are outside the "normal" range of outputs that you would obtain when handling your sequences one by one without padding. Thus, you may multiply this mask with the actual output of the convolutions to set all masked outputs to zero, and finally implement your own mean() operator that sums and divides by the number of unmasked timesteps.

That's not very easy to do ;-) and there may be other solutions I don't know.

user_26371260
Related