RandomizedSearchCV with scoring accuracy leads to error:Scoring failed.The score on this train-test partition for these parameters will be set to nan

user_16683404 0
Viewed 210

I want to find a good neural network instance by using RandomizedSearchCV with regard to accuracy, because the task is to solve a binary classification problem. Unfortunately I get the error message

Scoring failed. The score on this train-test partition for these parameters will be set to nan.

This is my implementation:

# Define neural network instance
def build_model(n_hidden_layers=2, n_neurons=77, dropout_rate=0.5 ,optimizer='adam', input_shape=77, activation_hidden="relu", activation_output="sigmoid",loss='binary_crossentropy',metrics=['binary_accuracy'],hidden_weight_initializer="he_normal",output_weight_initializer="glorot_normal",l1=0,l2=0,use_batch_norm=0):
    model = keras.models.Sequential()
    model.add(keras.layers.InputLayer(input_shape=input_shape))
    for layer in range(n_hidden_layers):
        model.add(keras.layers.Dense(n_neurons, activation=activation_hidden, kernel_initializer=hidden_weight_initializer, kernel_regularizer=tf.keras.regularizers.l1_l2(l1,l2)))
        model.add(keras.layers.Dropout(dropout_rate))
        if use_batch_norm == 1:
            model.add(keras.layers.BatchNormalization())
    model.add(keras.layers.Dense(1,activation=activation_output, kernel_initializer=output_weight_initializer))
    model.compile(loss=loss, optimizer=optimizer, metrics=metrics)
    return model

# Dreate wrapper class for RandomizedSearchCV
keras_reg = keras.wrappers.scikit_learn.KerasRegressor(build_model)

# Define hyperparameter spaces for trained neural network instances
param_distribs = {
    "n_hidden_layers": [1,2, 3,4,5],
    "n_neurons": [x for x in range(10,100)],
    "dropout_rate": [0, 0.1, 0.2, 0.3, 0.4, 0.5],
    "use_batch_norm": [0,1],            
   # "optimizer": ['adam',],
    "activation_hidden": ['relu','elu','selu'],# 'relu','','elu','selu',,'LeakyRelU(alpha=0.2)','PReLU(alpha_initializer=Constant(value=0.25))'
   # "activation_output": ['relu','sigmoid'],
   # "loss": ['binary_crossentropy']
   # "l1": 
   # "l2": 

}


from sklearn.metrics import make_scorer, precision_score, accuracy_score

precision = make_scorer(precision_score, pos_label="donated")
accuracy = make_scorer(accuracy_score, pos_label="donated")

# Use RandomizedSearchCV to find model instance with best performance on training data
rnd_search_cv = RandomizedSearchCV(keras_reg, param_distribs, n_iter=2, cv=2,scoring="accuracy")#, scoring=accuracy,random_state=1)#iter=10,cv=3
rnd_search_cv.fit(X_train, y_train, epochs=10,#100
                  validation_data=(X_test, y_test),
                  callbacks=[keras.callbacks.EarlyStopping(patience=5)],
                  batch_size=256)
gridsearchcv machine-learning neural-network python
1 Answers

I was trying to do the same thing and ran into the same problem. I found out that the problem was the scorer object supplied in RandomizedSearchCV(). The loss function should be specified within your build_model() function when compiling the model, and it must not be supplied by RandomizedSearchCV().

This should be easy if you want to use a simple loss function or metric like 'accuray' since it is already provided by keras. If you want a more specific loss function, you probably need to build it by yourself and make sure that it fulffils the requirements as described in the tensorflow documentation https://www.tensorflow.org/api_docs/python/tf/keras/Model.

user_139060010
Related