Convert tensor of size 768 to 128

You could achieve this using a wide flat kernel and/or combined with a specific stride. If you stick with a dilation of 1, then the input/output spatial dimension relation is given by:

out = [(2p + x - k)/s + 1]

Where p is the padding, k is the kernel size and s is the stride. [] detonates the whole part of the quantity.

Applied here you have:

128 = [(2p + 768 - k)/s + 1]

So you would get:

p = 2*p + 768 - (128-1)*s # one off

If you impose p = 0, and s = 6 you find k = 6

>>> project = nn.Conv2d(197, 197, kernel_size=(1, 6), stride=6)
>>> project(torch.rand(1, 197, 1, 768)).shape
torch.Size([1, 197, 1, 128])

Alternatively, a more straightforward - but different - approach is to learn a mapping using a fully connected layer:

>>> project = nn.Linear(768, 128)
>>> project(torch.rand(1, 197, 1, 768)).shape
torch.Size([1, 197, 1, 128])

You could use a kernel size and stride of 6, as that’s the factor between the input and output temporal size:

x = torch.randn(197, 1, 768)
conv = nn.Conv1d(in_channels=1, out_channels=1, kernel_size=6, stride=6)
out = conv(x)
print(out.shape)
> torch.Size([197, 1, 128])

Solution Source