How to pairwise combine a row and a column of features to a matrix of features in keras?

user_9883438 0
Viewed 365

In keras, using the functional API, I have two independent layers (tensors). The first one is a row vector of feature lists and the other one is a column vector of feature lists. For simplicity, suppose they are created like this:

rows = 5
cols = 10
features = 2

row = Input((1, cols, features))
col = Input((rows, 1, features))

Now I want to "merge" these two layers in a way that the result is a matrix with 5 rows and 10 columns (basically doing a 5x1 by 1x10 matrix multiplication) where each entry of that matrix is a concatenated feature list of every possible combination of the row and column vector. In other words, I'm looking for some MergeLayer that will combine my row and col layers to a matrix layer of shape (rows, cols, 2*features):

matrix = MergeLayer()([row, col]) # output_shape of matrix shall be (rows, cols, 2*features)

Example for cols = rows = 2:

row = [[[1,2]], [[3,4]]]
col = [[[5,6],
        [7,8]]]

matrix = [[[1,2,5,6], [3,4,5,6]],
          [[1,2,7,8], [3,4,7,8]]]

I'm assuming that the solution (if possible at all) will somehow leverage the the Dot layer and maybe some Reshape and/or Permute, but I can't figure it out.

keras python
1 Answers

You can repeat elements, then concatenate.

from keras.layers import Input, Lambda, Concatenate
from keras.models import Model
import keras.backend as K

rows = 2
cols = 2
features = 2

row = Input((1, cols, features))
col = Input((rows, 1, features))

row_repeated = Lambda(lambda x: K.repeat_elements(x, rows, axis=1))(row)
col_repeated = Lambda(lambda x: K.repeat_elements(x, cols, axis=2))(col)
out = Concatenate()([row_repeated, col_repeated])

model = Model(inputs=[row,col], outputs=out)
model.summary()

Experiment:

import numpy as np

x = np.array([1,2,3,4]).reshape((1, 1, 2, 2))
y = np.array([5,6,7,8]).reshape((1, 2, 1, 2))
model.predict([x, y])

#array([[[[1., 2., 5., 6.],
#         [3., 4., 5., 6.]],
#
#        [[1., 2., 7., 8.],
#         [3., 4., 7., 8.]]]], dtype=float32)
user_56803480
Related