I am implementing a neural network encoder for a (7,4) code. The theoretical upper bound for minimum distance of the code is 3 and Hamming code achieves it. So essentially I am trying to converge to Hamming code (or some permutation so that minimum distance is still 3). Following is how I setup the encoder in tensorflow:
def encoder(message):
weight1 = np.random.normal(loc=0.0, scale=0.01, size=[16, 32])
init1 = tf.constant_initializer(weight1)
out1 = tf.layers.dense(inputs=message, units=32, activation=None, kernel_initializer=init1)
weight2 = np.random.normal(loc=0.0, scale=0.01, size=[32, 16])
init2 = tf.constant_initializer(weight2)
out2 = tf.layers.dense(inputs=out1, units=16, activation=tf.nn.relu, kernel_initializer=init2)
weight3 = np.random.normal(loc=0.0, scale=0.01, size=[16, 7])
init3 = tf.constant_initializer(weight3)
out3 = tf.layers.dense(inputs=out2, units=7, activation=tf.nn.sigmoid, kernel_initializer=init3)
return out3
During training, to verify if the network I have created for encoder can really achieve minimum distance of 3, I use the ground truth, i.e., I feed one hot vector at the input of the neural network and the corresponding Hamming code vector for computing the error at the output. Using the following loss function, the network converges to Hamming code and achieves a minimum distance of 3.
codeword = encoder(input_bits)
loss = tf.reduce_mean(tf.multiply(0.5, tf.pow(tf.subtract(codeword, hamming_codeword), 2)))
Now this being verified, I change the loss function to the following and allow the encoder to try and find a code with as large minimum distance as possible.
codeword = encoder(input_bits)
distances = []
for i in range(0, 16):
for j in range(i+1, 16):
distances.append(tf.linalg.norm(codeword[i] - codeword[j]))
min_dist = tf.reduce_min(distances)
loss = - min_dist
But in this case I only get a best minimum distance of 2 or somewhere close to 2. How should the architecture be changed to achieve the expected minimum distance of 3? Thanks.