Confusing F1 score , and AUC scores in a highly imbalanced data while using 5-fold cross-validation

Question

I have been trying to classify the highly imbalanced data using 5-fold cross-validation. My sample size is:

Total samples: 12237899

Positive samples: 1064 (0.01% of total)

I wanted to avoid data leakage as well. However, I got pretty low average precision score, and F-1 scores. I have used the weighted logistic regression to help me with the imbalanced data since SMOTE doesn't work well in the presence of extremely imbalanced data. Also, I see a several options for F-1 score in the sklearn library. For example: f1 score has a argument like : average{‘micro’, ‘macro’, ‘samples’,’weighted’, ‘binary’}. Not sure which one I should use? Also, How it is different from scoring='f1' parameter of the cross_val_score(clf, X, y, cv=5, scoring='f1')?

from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import StratifiedKFold, train_test_split
from tqdm import tqdm
from sklearn.metrics import roc_auc_score, balanced_accuracy_score, f1_score, accuracy_score, confusion_matrix
from sklearn.metrics import roc_curve, auc
Balanced_Acc = []
F1 = []
G=[]
AP=[]
aucs = []
tprs = []
#fi = []
#rf_pi_train = []
#rf_pi_test = []
mean_fpr = np.linspace(0, 1, 100)
acc = []
cm = []
i=0
skf = StratifiedKFold(n_splits=5, random_state=None, shuffle=False)
for trainIndex, textIndex in tqdm(skf.split(X, y)):
    xTrain, xTest = X.iloc[trainIndex], X.iloc[textIndex]
    yTrain, yTest = y[trainIndex], y[textIndex]
    clf = LogisticRegression(class_weight='balanced',max_iter=100000)
    clf.fit(xTrain, yTrain)
    yPred = clf.predict(xTest)
    Balanced_Acc.append(balanced_accuracy_score(yTest, yPred))
    AP.append(average_precision_score(yTest, yPred))
    F1.append(f1_score(yTest,yPred))
    G.append(geometric_mean_score(yTest,yPred))
    #fi.append(clf.feature_importances_)
    #result_train = permutation_importance(clf, xTrain, yTrain, n_repeats=1)
    #result_test = permutation_importance(clf, xTest, yTest, n_repeats=1)
    #rf_pi_train.append(result_train.importances)
    #rf_pi_test.append(result_test.importances)

    acc.append(accuracy_score(yTest, yPred))
    cm.append(confusion_matrix(yTest,yPred))
    
    # ROC Curve
    fpr, tpr, thresholds = roc_curve(yTest, yPred)
    tprs.append(interp(mean_fpr, fpr, tpr))
    tprs[-1][0] = 0.0
    roc_auc = auc(fpr, tpr)
    aucs.append(roc_auc)
    plt.plot(fpr, tpr, lw=1, alpha=0.3,
             label='ROC fold %d (AUC = %0.2f)' % (i, roc_auc))
    i = i+1
    
plt.plot([0, 1], [0, 1], linestyle='--', lw=2, color='r', label='Chance', alpha=.8)

mean_tpr = np.mean(tprs, axis=0)
mean_tpr[-1] = 1.0
mean_auc = auc(mean_fpr, mean_tpr)
std_auc = np.std(aucs)
plt.plot(mean_fpr, mean_tpr, color='b',
         label=r'Mean ROC (AUC = %0.2f $\pm$ %0.3f)' % (mean_auc, std_auc),
         lw=2, alpha=.8)

std_tpr = np.std(tprs, axis=0)
tprs_upper = np.minimum(mean_tpr + std_tpr, 1)
tprs_lower = np.maximum(mean_tpr - std_tpr, 0)
plt.fill_between(mean_fpr, tprs_lower, tprs_upper, color='grey', alpha=.2,
                 label=r'$\pm$ 1 std. dev.')

plt.xlim([-0.05, 1.05])
plt.ylim([-0.05, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver operating characteristic')
plt.legend(loc="lower right")
plt.show()
    
# print(cm[0])
tp = fp = fn = tn = 0
for m in cm:
    tp += m[0][0]
    fp += m[0][1]
    
    fn += m[1][0]
    tn += m[1][1]
    
# print(tp, fp, fn, tn)
finalCM = [[tp, fp], [fn, tn]]

print(finalCM)
ax = sns.heatmap(finalCM, annot=True, cbar=False, fmt='g')
bottom, top = ax.get_ylim()
ax.set_ylim(bottom + 0.5, top - 0.5)
plt.ylabel('True Label')
plt.xlabel('Predicted Label')
plt.title('Confusion Matrix')

print("Balanced Accuracy: ", np.mean(Balanced_Acc))
print("AP score: ", np.mean(AP))
print("G-mean: ", np.mean(G))
print("F1: ", np.mean(F1))
print('AUC: ', np.mean(aucs))
#AUC_rf = aucs

I am not sure why I am seeing the balanced accuracy and AUC scores are same!I appreciate your thoughts! Thanks!

user_9632441 · Answer

You are actually asking three separate questions:

Why is ROC AUC and Balanced Accuracy so high?
Why is Average Precision and F1 score so low?
Which F1 score is appropriate for imbalanced classification?

Reminder

Sensitivity equation: sensitivity = TP / (TP + FN)

False Positive Rate equation: FPR = FP / (FP + TN)

Specificity equation: specificity = 1 - FPR

In case of positive class imbalance, TN in FPR is the main culprit.

Let's look at the simulated example:

from sklearn.metrics import classification_report
import numpy as np

y_true = np.concatenate([np.ones(10), np.zeros(99990)])
y_pred = np.concatenate([np.zeros(9), np.ones(1), np.zeros(99990)])
print(classification_report(y_true, y_pred))

Which outputs this:

              precision    recall  f1-score   support

         0.0       1.00      1.00      1.00     99990
         1.0       1.00      0.10      0.18        10

    accuracy                           1.00    100000
   macro avg       1.00      0.55      0.59    100000
weighted avg       1.00      1.00      1.00    100000

Sensitivity in binary classification case is the recall of a positive class, hence 0.1.

Similarly, specificity is the recall of a negative class, hence 1.0.

FPR is 1 - sensitivity = 1 - 0.1 = 0.9.

What's the problem?

ROC AUC

ROC AUC calculates is the sum of sensitivity weighted by FPR for all possible thresholds. Since FPR is inflated due to highly imbalanced negative class, model gets high ROC AUC score without much effort.

Balanced Accuracy

Now, that we understand that, it should be clear why Balanced Accuracy is also very high. Look at the equation: balanced accuracy = mean(specificity, sensitivity). Since specificity is inflated, simple average is also biased towards the majority class.

Ok, how to fix it?

Balanced Accuracy can be adjusted to classes imbalance by specifying adjusted=True in sklearn.metrics.balanced_accuracy_score. As for ROC AUC, the alternative is to use Precision-Recall AUC, which is exactly sklearn.metrics.average_precision_score.

What about f1 score options?

Default for binary classification is to calculate f1 score for positive class only. As described in documentation, default is average='binary'.

Let's compare all average options on our synthetic example:

f1_score(y_true, y_pred, average='binary')   # 0.1818...
f1_score(y_true, y_pred, average='micro')    # 0.9991...
f1_score(y_true, y_pred, average='macro')    # 0.5908...
f1_score(y_true, y_pred, average='weighted') # 0.9998...

(None returns a tuple of f1 scores for positive and negative classes, while 'samples' is not applicable in our case)

Reminder is relevant:

Precision equation: precision = TP / (TP + FP)

Recall equation: recall = TP / (TP + FN)

f1 score: f1_score = 2 * precision * recall / (precision + recall)

Since it doesn't take into account TN, default f1 score is ignoring model ability to successfully detect the majority class. This may be too harsh in some circumstances, therefore other options try to include that into account using different strategies:

average="micro" counts TP, FP, FN for positive and negative classes, adds them up, and calculates precision, recall, f1.
average="macro" counts TP, FP, FN, and calculates f1 separately for each class, and calculates unweighted average of all f1 scores
average="weighted" does average="macro", but with weighted average by support (ie, number of samples for each class)

Which f1 score to choose depends heavily on application. From my experience, average="binary" was too harsh on the model performance, but I haven't had as severe class imbalance, as you are.

In your case, AP and F1 score are so low because model cannot successfully predict positive class. There are plenty of strategies, I'll suggest something that worked for me: selecting a representative but much smaller subset of the majority class.

There are plenty of methods on Instance Selection, Selective Nearest Neighbors and Iterative Case Filtering to name a few. I've found this article very informative.