Multi-class loss keeps increasing XGBoost

Question

I have been trying to implement the gradient and hessian of this loss:

In python, through this code (I've highlighted the terms in order to be as much clear as possible):

def guess_averse_loss(pred, dtrain):
    
    labels_indexes = dtrain.get_label().astype(int)
    labels = np.eye(pred.shape[1])[labels_indexes]

    # Array C_z for each sample
    penality_vector = np.dot(labels, COST_MATRIX) 
    # Array containing score of class z for each sample
    z_score_vector = np.repeat(pred[np.arange(pred.shape[0]), labels_indexes], pred.shape[1]).reshape(pred.shape)
    # C_z,i * exp(S_i - S_z) (for all classes, for all samples)
    first_term = np.multiply(penality_vector, np.exp((pred-z_score_vector)))
    # Sum over all classess of the C_z,i * exp(S_i - S_z) term
    loss_term = np.sum(first_term, axis=1)
    # 1{z=i}*A
    second_term = np.multiply(labels, loss_term[:, np.newaxis])

    # This is the gradient of A, simply C_z,i * exp(S_i - S_z) - 1{z=i}*A
    grad_norm = (first_term - second_term)
    # Divide for (1 + A)
    grad = grad_norm/(1+loss_term[:, np.newaxis])

    # IT = C_z,i * exp(S_i - S_z) + partial(A)/partial(S_i)
    internal_term = first_term+grad_norm
    # IT2 = C_z,i * exp(S_i - S_z) - 1{z=i}*IT
    first_hessian_term = first_term-np.multiply(labels, internal_term)
    # (1+A)*IT2
    first_hessian_term = first_hessian_term * (1+loss_term[:, np.newaxis])
    # partial(A)/partial(S_i)*partial(A)/partial(S_i)
    second_hessian_term = grad_norm**2

    # Numerator hessian of loss: 
    hess_norm = first_hessian_term-second_hessian_term
    # Divide for (1+A)^2
    hess = hess_norm / ((1+loss_term[:, np.newaxis])**2)

    grad = grad.reshape((pred.shape[0]*pred.shape[1], 1))
    hess = hess.reshape((pred.shape[0]*pred.shape[1], 1))

    #print(grad, hess)

    return grad, hess

Consider dtrain and pred to be (num_prediction,) and (num_prediction,num_classes) vectors with real labels (the first) and predicted scores for each class (the second).

I suspect that I made some implementation errors, since the loss keeps increasing when using it in xgboost. I didn't succeed in finding the error. I would like to know if this is the correct way to proceed or there is some error in the process I can't catch.

The loss is doing something like that:

Multi-class loss keeps increasing XGBoost

0 Answers