Multi-branch neural network does not learn dependencies

Viewed 87

I am currently trying to learn using a multi-branch neural network. The underlying data looks something like the following table:

Var_1 Var_2 Var_3
a_1 a_2 a_3
a_1 a_2 c_3
b_1 b_2 b_3
b_1 b_2 a_3
a_1 a_2 b_3

I would now expect that my neural network learns besides the distribution of variables per column (Var_1: a_1=3/5, b_1=2/5; Var_2: a_2=3/5, b_2=2/5; Var_3: a_3=2/5, b_3=2/5; c_3=1/5) also that there is an dependency between the occurance of a_1 and a_2 as well as b_1 and b_2.

But my observation is that the neural network only learns the distribution of the variables per columns and then always predicts the most often occuring value per column. So if I would like to predict Var_1 and Var_3 have already given Var_2 the predictions are indifferent of what value for Var_2 is already given. It always predicts the most often occuring value per column and looks the following:

Var_2=a_2 -> Var_1=a_1; Var_3=a_3

Var_2=b_2 -> Var_1=a_1; Var_3=a_3

The neural network is built the following:

inputs = tf.keras.Input(shape=(input_shape,), name="Configuration_data")
        x = tf.keras.layers.Dense(input_shape, activation=tf.nn.relu)(inputs)
        y = tf.keras.layers.Dense(input_shape, activation=tf.nn.relu)(x)
        z = tf.keras.layers.Dense(input_shape, activation=tf.nn.relu)(y)
        outputs = []
        for label_name, labels in label_dict.items():
            output_shape = len(labels)
            outputs.append(ConLearn.build_branch(input_shape, output_shape, z, label_name))

        model = tf.keras.models.Model(inputs=inputs, outputs=outputs, name="ConLearn")

        return model

    def build_branch(input_shape, output_shape, x, label_name):
        if output_shape < 3:
            x = tf.keras.layers.Dense(output_shape, activation=tf.nn.sigmoid, name=label_name)(x)
        else:
            x = tf.keras.layers.Dense(output_shape, activation=tf.nn.softmax, name=label_name)(x)

        return x

Here a snapshot of the network: Multi-branch neural network

I use Adam as an optimizer, but have already tried some others.

Edit: I recognized in the meantime that the last branch is learning correctly. I changed the variable that is in the last branch (most right in the picture of the output layer) to verify that and it has proven right. The accuracy is continuously increased for this variable during training whereas the other variables stop at the proportion of the most often occurring variable value. A very strange behavior for my understanding.

Any recommendation would be highly welcome. Thank you!

Best regards, Mathias

0 Answers
Related