Using a Custom Dual Input Keras Layer

I am attempting to create a simple multi-layer perceptron in Keras. The general structure I would like to create is one where a matrix A of dimension [n_a1, n_a2] is sent through a number of layers of a multilayer perceptron, and at a certain point, the dot product of the morphed A matrix is taken with a randomly selected y vector [n_y, 1] from a set of y vectors, and the result then continues through a number more layers before reaching the end where it is compared with the input labels as normal.

Unfortunately, I am having issues with figuring how to implement this properly. I created a custom multi-input layer per the simple example offered at https://keras.io/layers/writing-your-own-keras-layers/, but it seems that I still can't have multiple inputs in the network. I am getting an error for setting an array element as a sequence: ValueError: setting an array element with a sequence.

Also, its unclear to me how the network would know what to do with the list formatted inputs for the other layers. Do I need to specify the list shape for each layer, and to somehow only use the A in the [A, y] list?

Custom Layer

class DualLayer(Layer):

    def __init__(self, output_dim, **kwargs):
        self.output_dim = output_dim
        super(DualLayer, self).__init__(**kwargs)

    def build(self, input_shape):
        #Trainable weight variable for layer
        self.kernel = self.add_weight(name='kernel',
                                     shape=(input_shape[1], self.output_dim),
                                     initializer='uniform',
                                     trainable=True)
        super(DualLayer, self).build(input_shape)

    def call(self, x):
        aOpt, y = x
        return [K.dot(aOpt, y)]

    def compute_outpute_shape(self, input_shape):
        assert isinstance(input_shape, list)
        shape_y, shape_aOpt = input_shape
        return [shape_y[0]]

Model

def modFunc(y1, A1, y2, A2, xSim):
    model = Sequential()
    model.add(Flatten(input_shape = np.shape(A1)))
    model.add(Dense(np.shape(A1)[0]*np.shape(A1)[1], activation = 'relu',
                    kernel_regularizer = 'l1', activity_regularizer = 'l1'))
    model.add(DualLayer([y1, A1]))
    model.add(Dense(5, activation = 'relu'))

    clf = model.fit([y1, A1], xSim, epochs=5, batch_size=1, verbose=2)
    return clf.coef_