Based on the Tensorflow Documentation, one can add label smoothing to categorical_crossentropy by adding label_smoothing argument. My question is what about sparse categorical crossentropy loss. There is no label_smoothing argument for this loss function.
3 Answers
It's easy to write your own loss function:
from tensorflow.keras.losses import categorical_crossentropy
def scce_with_ls(y, y_hat):
y = tf.one_hot(tf.cast(y, tf.int32), n_classes)
return categorical_crossentropy(y, y_hat, label_smoothing = 0.1)
There is no label_smoothing argument when we are using sparse categorical cross entropy as loss function.
there is a better way if you use softmax in the last activation than label smoving by t is the same as using t/num_classes l1 regulerization (before softmax) with weight 1/(num/t-2) and then scale by 1-2t/num.
the math is kinda simple importent trick is to take the lse(x) out of the log and then it cancels out nicely with everything leaving you with only the logs of x scaled by labels
Related