trying to callibrate keras model

Question

I'm trying to calibrate my CNN model by Sklearn implementation CalibratedClassifierCV, tried to wrap it as KerasClassifier and to override the predict function but without success. someone could say me what I did wrong? this is the model code:

def create_model():
    model = Sequential()
    model.add(Conv2D(64, kernel_size=(3,3), activation = 'relu', input_shape=(28, 28 ,1) ))
    model.add(MaxPooling2D(pool_size = (2, 2)))

    model.add(Conv2D(64, kernel_size = (3, 3), activation = 'relu'))
    model.add(MaxPooling2D(pool_size = (2, 2)))

    model.add(Conv2D(64, kernel_size = (3, 3), activation = 'relu'))
    model.add(MaxPooling2D(pool_size = (2, 2)))

    model.add(Flatten())
    model.add(Dense(128, activation = 'relu'))
    model.add(Dropout(0.20))

    model.add(Dense(24, activation = 'softmax'))

    model.compile(loss = keras.losses.categorical_crossentropy, optimizer=keras.optimizers.Adam(), metrics=['accuracy'])
    
    return model

this is me trying to calibrate it :

model = KerasClassifier(build_fn=create_model,epochs=5, batch_size=128,validation_data=(evalX_cnn, eval_y_cnn))
model.fit(trainX_cnn, train_y_cnn)
model_c = CalibratedClassifierCV(base_estimator=model, cv='prefit')
model_c.fit(valX_cnn, val_y_cnn)

the output :

-------------------------------------------------------
TypeError             Traceback (most recent call last)
 in 
----> 1 model_c.fit(np.array(valX_cnn), np.array(val_y_cnn))

~\anaconda3\lib\site-packages\sklearn\calibration.py in fit(self, X, y, sample_weight)
    286             pred_method, method_name = _get_prediction_method(base_estimator)
    287             n_classes = len(self.classes_)
--> 288             predictions = _compute_predictions(pred_method, method_name, X, n_classes)
    289 
    290             calibrated_classifier = _fit_calibrator(

~\anaconda3\lib\site-packages\sklearn\calibration.py in _compute_predictions(pred_method, method_name, X, n_classes)
    575         (X.shape[0], 1).
    576     """
--> 577     predictions = pred_method(X=X)
    578 
    579     if method_name == "decision_function":

TypeError: predict_proba() missing 1 required positional argument: 'x'

valX_cnn and val_y_cnn are of type np.array.

tried even to override the method:

keras.models.Model.predict_proba = keras.models.Model.predict

user_10375049 · Answer

The problem is because predict_proba from KerasClassifier requires x as input while predict_proba method from sklearn accepts X as input argument (note the difference: X is not x).

You can simply overdrive the problem wrapping KerasClassifier into a new class to correct the predict_proba method.

samples,classes = 100,3

X = np.random.uniform(0,1, (samples,28,28,1))
Y = tf.keras.utils.to_categorical(np.random.randint(0,classes, (samples)))

class MyKerasClassifier(KerasClassifier):
  
  def predict_proba(self, X):
      return self.model.predict(X)

model = MyKerasClassifier(build_fn=create_model, epochs=3, batch_size=128)
model.fit(X, Y)

model_c = CalibratedClassifierCV(base_estimator=model, cv='prefit')
model_c.fit(X, Y)

user_5980568 · Answer

The wrappers are going to be deprecated. From tensorflow>=2.7.0 you might want to use scikeras.

pckg repo>
https://github.com/adriangb/scikeras

code example>

from sklearn.datasets import make_classification
from tensorflow import keras
from scikeras.wrappers import KerasClassifier


X, y = make_classification(1000, 20, n_informative=10, random_state=0)
X = X.astype(np.float32)
y = y.astype(np.int64)

def get_model(hidden_layer_dim, meta):
    # note that meta is a special argument that will be
    # handed a dict containing input metadata
    n_features_in_ = meta["n_features_in_"]
    X_shape_ = meta["X_shape_"]
    n_classes_ = meta["n_classes_"]

    model = keras.models.Sequential()
    model.add(keras.layers.Dense(n_features_in_, input_shape=X_shape_[1:]))
    model.add(keras.layers.Activation("relu"))
    model.add(keras.layers.Dense(hidden_layer_dim))
    model.add(keras.layers.Activation("relu"))
    model.add(keras.layers.Dense(n_classes_))
    model.add(keras.layers.Activation("softmax"))
    return model

clf = KerasClassifier(
    get_model,
    loss="sparse_categorical_crossentropy",
    hidden_layer_dim=100,
)

clf.fit(X, y)
y_proba = clf.predict_proba(X)

trying to callibrate keras model

2 Answers