I am currently trying to learn using a multi-branch neural network. The underlying data looks something like the following table:
| Var_1 | Var_2 | Var_3 |
|---|---|---|
| a_1 | a_2 | a_3 |
| a_1 | a_2 | c_3 |
| b_1 | b_2 | b_3 |
| b_1 | b_2 | a_3 |
| a_1 | a_2 | b_3 |
I would now expect that my neural network learns besides the distribution of variables per column (Var_1: a_1=3/5, b_1=2/5; Var_2: a_2=3/5, b_2=2/5; Var_3: a_3=2/5, b_3=2/5; c_3=1/5) also that there is an dependency between the occurance of a_1 and a_2 as well as b_1 and b_2.
But my observation is that the neural network only learns the distribution of the variables per columns and then always predicts the most often occuring value per column. So if I would like to predict Var_1 and Var_3 have already given Var_2 the predictions are indifferent of what value for Var_2 is already given. It always predicts the most often occuring value per column and looks the following:
Var_2=a_2 -> Var_1=a_1; Var_3=a_3
Var_2=b_2 -> Var_1=a_1; Var_3=a_3
The neural network is built the following:
inputs = tf.keras.Input(shape=(input_shape,), name="Configuration_data")
x = tf.keras.layers.Dense(input_shape, activation=tf.nn.relu)(inputs)
y = tf.keras.layers.Dense(input_shape, activation=tf.nn.relu)(x)
z = tf.keras.layers.Dense(input_shape, activation=tf.nn.relu)(y)
outputs = []
for label_name, labels in label_dict.items():
output_shape = len(labels)
outputs.append(ConLearn.build_branch(input_shape, output_shape, z, label_name))
model = tf.keras.models.Model(inputs=inputs, outputs=outputs, name="ConLearn")
return model
def build_branch(input_shape, output_shape, x, label_name):
if output_shape < 3:
x = tf.keras.layers.Dense(output_shape, activation=tf.nn.sigmoid, name=label_name)(x)
else:
x = tf.keras.layers.Dense(output_shape, activation=tf.nn.softmax, name=label_name)(x)
return x
Here a snapshot of the network:

I use Adam as an optimizer, but have already tried some others.
Edit: I recognized in the meantime that the last branch is learning correctly. I changed the variable that is in the last branch (most right in the picture of the output layer) to verify that and it has proven right. The accuracy is continuously increased for this variable during training whereas the other variables stop at the proportion of the most often occurring variable value. A very strange behavior for my understanding.
Any recommendation would be highly welcome. Thank you!
Best regards, Mathias