Implementing Cosine similarity loss gives different answer than Tensorflow's

Admin 1
Viewed 509

I was implementing cosine similarity loss with my custom python script but it gives me a very different answer than TensorFlow. First see TensorFlow's answer:-

y_true = [[0., 1.], [1., 1.]]
y_pred = [[0., 1.], [0., 1.]]
loss = tf.keras.losses.CosineSimilarity()
print(loss(y_true, y_pred).numpy())

Output:

>>> -0.8535534

According to the TensorFlow documentation, the formula to compute the loss is this:-

enter image description here

I implemented the same with plain python as this:-

def cosine_similarity(y_true, y_pred):
    loss = -np.sum(np.linalg.norm(y_true) * np.linalg.norm(y_pred))
    return loss

print(cosine_similarity(y_true, y_pred))

Output:

>>> -2.4494897427831783

I don't why I am getting -2.45 and TensorFlow is outputting -0.85. Any solution so my answer can match with TensorFlow's?

keras tensorflow python
1 Answers

After going through some documentation,

results from tf.keras.losses.CosineSimilarity()and your function differs for two reasons:

  1. As presented in the example here, in CosineSimiliraty() function, L2_normalisation is done along axis=1

When np.linalg.norm() is performed on the whole array because no axis was given. Moreover, the result is summed.

y_true = [[0., 1.], [1., 1.]]
y_pred = [[0., 1.], [0., 1.]]

print(tf.math.l2_normalize(y_true,axis=1))

print(np.linalg.norm(y_true))


Outputs
#[[0.         1.        ]
# [0.70710677 0.70710677]]
# 1.7320508075688772 
# Result from np.linalg.norm() is obtained by summing  : 
#[[0.         0.57735026]
#[0.57735026  0.57735026]]
  1. Secondly I don't know why but considering the example given in the link above, Before summing values, np.mean is applied along the same axis. They might forget to precise it, in the formula you used.
a=tf.math.l2_normalize(y_true,axis=1)
b=tf.math.l2_normalize(y_pred,axis=1)

print(a)
print(b)
print(np.mean(a*b,axis=1)
print(-np.sum(np.mean(a*b,axis=1)))

#[[0. 1.][0.70710677 0.70710677]]
#[[0. 1.][0. 1.]]
#[0.5        0.35355338]
# -0.8535534
#
  1. I am not sure but np.linalg.norm() seems to give the norm of a vector/matrix Where tensorflow gives you the same matrix normalized (along the axis you selected)

So instead of using np.linalg.norm() use Tensorflow function tf.math.l2_normalized(myarray,axis=1)


def cosine_sim(y_true,y_pred):
    norm_true=tf.math.l2_normalize(y_true,axis=1)
    norm_pred=tf.math.l2_normalize(y_pred,axis=1)

    loss =-np.sum(np.mean(norm_true*norm_pred,axis=1))
    return loss
user_140467980
Related