Intro
Hej everyone,
I am working on my diploma thesis and I face a binary classification problem with imbalanced class contribution. I have around 10 times more negative ("0") labels as positive ("1") labels. For that reason I considered not only observing accuracy and ROC-AUC, but also weighted/ balanced accuracy and Precision-Recall-AUC.
I already asked the question on GitHub (https://github.com/keras-team/keras/issues/12991) but the issue has not been answered yet so I thought this platform here might be the better place!
Issue description
During some calculations on the validation set in a custom callback I noticed, more or less by coincidence, that the weighted accuracy is always different from my results using sklearn.metrics.accuracy_score().
Using Keras, weighted accuracy has to be declared in model.compile() and is a key in the logs{} dictionary after every epoch (and is also written to the log file by the CSVLogger callback or to the history object) or is returned as value in a list by model.evaluate(),
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'],
weighted_metrics=['accuracy'])
I calculate the val_sample_weights vector based on the class contribution of the training set with the Sklearn.metrics function class_weight.compute_sample_weight() and with the help of class_weight.compute_class_weight().
cls_weights = class_weight.compute_class_weight('balanced', np.unique(y_train._values),
y_train._values)
cls_weight_dict = {0: cls_weights[0], 1: cls_weights[1]}
val_sample_weights = class_weight.compute_sample_weight(cls_weight_dict, y_test._values)
In model.fit() I pass this vector togehter with the validation data and to sklearn.metrics.accuracy_score() I pass it to the parameter name sample_weight to compare the results on the same basis.
model_output = model.fit(x_train, y_train, epochs=500, batch_size=32, verbose=1,
validation_data=(x_test, y_test, val_sample_weights))
Furthermore, I derived the equation how Scitkit-learn computes the weighted accuracy from several easy examples and it seems that it's computed by the following equation (which seems quite reasonable to me):
TP, TN, FP and FN are the values reported in the confusion matrix and w_p and w_n are the class weights of the positive and negative class respectively.
An easy example to test it can be found here:
https://scikit-learn.org/stable/modules/generated/sklearn.metrics.balanced_accuracy_score.html
Just for the sake of completeness, sklearn.metrics.accuracy_score(..., sample_weight=) returns the same result as sklearn.metrics.balanced_accuracy_score().
System Information
- GeForce RTX 2080 Ti
- Keras 2.2.4
- Tensorflow-gpu 1.13.1
- Sklearn 0.19.2
- Python 3.6.8
- CUDA Version 10.0.130
Code example
I searched an easy example to make the issue easy to reproduce, even if the class imbalance here is weaker (1:2 not 1:10). It's based on the introductory tutorial to Keras which can be found here:
https://towardsdatascience.com/k-as-in-keras-simple-classification-model-a9d2d23d5b5a
The Pima Indianas onset diabets dataset will be downloaded, as done in the link above, from the repository of Jason Brownlee, the maker of the homepage Machine Learning Mastery. But I guess it can also be downloaded from various other sites.
So finally here's the code:
from keras.layers import Dense, Dropout
from keras.models import Sequential
from keras.regularizers import l2
import pandas as pd
import numpy as np
from sklearn.utils import class_weight
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split
file = 'https://raw.githubusercontent.com/jbrownlee/Datasets/master/' \
'pima-indians-diabetes.data.csv'
# Load csv data from file to data using pandas
data = pd.read_csv(file, names=['pregnancies', 'glucose', 'diastolic', 'triceps', 'insulin',
'bmi', 'dpf', 'age', 'diabetes'])
# Process data
data.head()
x = data.drop(columns=['diabetes'])
y = data['diabetes']
# Split into train and test
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.1, random_state=0)
# define a sequential model
model = Sequential()
# 1st hidden layer
model.add(Dense(100, activation='relu', input_dim=8, kernel_regularizer=l2(0.01)))
model.add(Dropout(0.3))
# 2nd hidden layer
model.add(Dense(100, activation='relu', kernel_regularizer=l2(0.01)))
model.add(Dropout(0.3))
# Output layer
model.add(Dense(1, activation='sigmoid'))
# Compilation with weighted metrics
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'],
weighted_metrics=['accuracy'])
# Calculate validation _sample_weights_ based on the class distribution of train labels and
# apply it to test labels using Sklearn
cls_weights = class_weight.compute_class_weight('balanced', np.unique(y_train._values),
y_train._values)
cls_weight_dict = {0: cls_weights[0], 1: cls_weights[1]}
val_sample_weights = class_weight.compute_sample_weight(cls_weight_dict, y_test._values)
# Train model
model_output = model.fit(x_train, y_train, epochs=500, batch_size=32, verbose=1,
validation_data=(x_test, y_test, val_sample_weights))
# Predict model
y_pred = model.predict(x_test, batch_size=32, verbose=1)
# Classify predictions based on threshold at 0.5
y_pred_binary = (y_pred > 0.5) * 1
# Sklearn metrics
sklearn_accuracy = accuracy_score(y_test, y_pred_binary)
sklearn_weighted_accuracy = accuracy_score(y_test, y_pred_binary,
sample_weight=val_sample_weights)
# metric_list has 3 entries: [0] val_loss weighted by val_sample_weights, [1] val_accuracy
# [2] val_weighted_accuracy
metric_list = model.evaluate(x_test, y_test, batch_size=32, verbose=1,
sample_weight=val_sample_weights)
print('sklearn_accuracy=%.3f' %sklearn_accuracy)
print('sklearn_weighted_accuracy=%.3f' %sklearn_weighted_accuracy)
print('keras_evaluate_accuracy=%.3f' %metric_list[1])
print('keras_evaluate_weighted_accuracy=%.3f' %metric_list[2])
Results and summary
For example I get:
sklearn_accuracy=0.792
sklearn_weighted_accuracy=0.718
keras_evaluate_accuracy=0.792
keras_evaluate_weighted_accuracy=0.712
The "unweighted" accuracy value is the same, both for Sklearn as for Keras. The difference isn't really big, but it grows bigger as the dataset becomes more imbalanced. For example for my task it always differs around 5% from each other!
Maybe I'm missing something and it's supposed to be like that, but anyways it's confusing that Keras and Sklearn provide different values, especially thinking of the whole class_weights and sample_weights thing as a topic hard to get into. Unfortunately I'm not too deep into Keras to search in the Keras code on my own.
I would really appreciate receiving any answers!