How can I access the best neural network model identifed through RandomizedSearchCV

Viewed 112

I am looking to extract and use (within the same Jupyter notebook) the model identified as the best model from RandomizedSearchCV for future fitting and graphing. Specifically, I am looking to re-fit the Keras Neural Network identified as the best so that I can plot the loss and accuracy against the same or other dataset.

If I run the following code, I get the output I expect - the best score and the paramaters used in obtaining that score.

import pandas as pd
from sklearn.preprocessing import LabelEncoder
from sklearn.pipeline import Pipeline
from skrebate import SURF
from sklearn.model_selection import StratifiedKFold
from sklearn.model_selection import GridSearchCV, RandomizedSearchCV
from keras.wrappers.scikit_learn import KerasClassifier
from keras.models import Sequential
from keras.layers import Dense


url = "https://datahub.io/machine-learning/sonar/r/sonar.csv"
dataframe = pd.read_csv(url)

dataset = dataframe.values

X = dataset[:,0:60].astype(float)
y = dataset[:,60]

features, labels = dataset[:,0:60].astype(float), dataset[:,60]

encoder = LabelEncoder()
encoder.fit(y)
encoded_y = encoder.transform(y)



def create_nn_model(input_dims):
    # Create model.
    model = Sequential()
    model.add(Dense(60, input_dim=input_dims, activation='relu'))
    model.add(Dense(1, activation='sigmoid'))
    # Compile model.
    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
    return model

param_grid = {'model__epochs': (100,200,300),
         'model__batch_size': (10,20,20)}

kfold = StratifiedKFold(n_splits=10, shuffle=True)

for x in range(10, 11, 10): # Iterate through top 10% to 100% of the features.

    num_features = int(features.shape[1] * x / 100)

    clf = Pipeline([('fs_step', SURF(n_features_to_select=num_features)),
                    ('model', KerasClassifier(build_fn=lambda: create_nn_model(num_features), epochs=100, batch_size=5, verbose=0))])

    grid = RandomizedSearchCV(clf, param_grid, n_jobs=-1, cv=kfold, n_iter = 3)
    grid_result = grid.fit(features, encoded_y)

    print('Best score obtained: {0}'.format(grid_result.best_score_))

    print('Parameters:')
    for param, value in grid_result.best_params_.items():
        print('\t{}: {}'.format(param, value))

I also know how to plot the data I am looking for by building, compiling and fitting a model, as per the following code:

model = Sequential()
model.add(Dense(60, input_dim=60, activation='relu'))
model.add(Dense(1, activation='sigmoid'))
# Compile model
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

model.fit(features, encoded_y, epochs=100, batch_size=25, verbose=0)

losses = pd.DataFrame(model.history.history)
import matplotlib.pyplot as plt

losses.plot()
plt.show()

Can I take the best model from the RandomizedSearchCV, fit against data and plot, or must I build, compile and fit based on best_params_? The reason I ask is because I can not identify / access what I understood to be the best model with grid_result.best_estimator_.model. Trying to do so gives:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-31-bbfe0b584f46> in <module>
----> 1 grid_result.best_estimator_.model

AttributeError: 'Pipeline' object has no attribute 'model'

Any help would be appreciated. Thanks.

1 Answers

grid_result.best_estimator_ contains the refit estimator (since you've left the default value for the refit parameter), which is a fitted clone of your clf. That happens to be a pipeline object (with two steps) in your case; if you want to access the keras model, you can access it as though a dictionary:

grid_result.best_estimator_['model'] will be a fitted KerasClassifier object. And those have the model attribute which contains the native keras object:

grid_result.best_estimator_['model'].model

Related