Concatenate two tensors of different shape from two different input modalities

user_14557136 0
Viewed 430

I have two tensors:

a = torch.randn((1, 30, 1220)) # represents text embedding vector (30 spans, each with embedding size of 1220)
b = torch.randn((1, 128, 256)) # represents image features obtained from a pretrained CNN (object detection)

  1. How do I concatenate everything in b to each one of the 30 spans of a?

  2. How to concatenate the whole b to the whole a?

This is what I'm trying to do:

Image features concatenated with LSTM output

The authors have only provided this text:

Model description

I'm extracting features (outlined in red) from a 3d point cloud (similar to CNN but for 3d) as shown below:

3D Feature Extraction

pytorch deep-learning machine-learning concatenation python
2 Answers

You're looking to combine two tensors with different shapes, there is no trivial way of concatenating them. Both tensors hold information regarding the same instance: the element you want to characterize with features embeddings through two different modalities: textual and visual.

The only way that makes sense to me is to learn two separate layers to map your text embedding and your image features to a common space where you can easily fuse them.

The design you adopt for this mapping is entirely up to you. Of course, this mapping layers need to be learned through training i.e. applying some kind of supervision at the other end.

user_63313690

Since these representations are from two different modalities (i.e., text and image) and they contain valuable features that are of great importance to the final goal, I would suggest to fuse them in a "learnable" manner instead of a mere concatenation or addition. Furthermore, such a learnable weighting (between features) would be optimal since in some cases one representation would be far more useful than the other whereas at other instances the vice versa applies.

Please note that a mere concatenation can also happen in this fusion module that you would implement. For the actual implementation, there are several types of fusion techniques. E.g. Simple Fusion, Cold Fusion etc. (cf. Fusion Models for Improved Visual Captioning, 2021)

For instance, one straightforward idea would be to use a simple linear layer to project one of the features to the same dimensionality as the other and then do a simple concatenation, with some optional non-linearity if needed.

user_29560660
Related