Optimize number of hidden layers and neurons with RandomizedSearchCV (scikit-learn) -> No unnecessary trainings?

user_14678325 0
Viewed 739

I want to optimize the number of hidden layers and the number of units in each hidden layer. For this I used RandomizedSearchCV from sklearn in this way:

    from tensorflow.keras.wrappers.scikit_learn import KerasRegressor
    from sklearn.model_selection import RandomizedSearchCV
    from tensorflow import keras
    models = keras.models
    layers = keras.layers

    def tuneModel(n_layer, n1_units, n2_units):
        model = models.Sequential()

        # input layer + first hidden layer
        model.add(layers.Dense(n1_units, input_dim=100, activation="relu"))       
        
        # second hidden layer
        if n_layer == 2:
            model.add(layers.Dense(n2_units, activation="relu"))   
    
        # output layer
        model.add(layers.Dense(2))                                                  
        
        model.compile(loss="mse", optimizer=opt)
        return model
    
    model = KerasRegressor(build_fn=tuneModel, verbose=0)
    
    param_grid = dict(n_layer=[1,2],
                      n1_units=[16, 32, 64, 128],                  
                      n2_units=[16, 32, 64, 128],
                      batch_size=[64],
                      epochs=[100]
                      )

    grid = RandomizedSearchCV(estimator=model,
                              param_distributions=param_grid, 
                              scoring="neg_mean_squared_error",
                              cv=5,
                              n_iter=8
                              )
    
    grid_result = grid.fit(x,y)

This is a strong downsized example for showing my problem. It works well, but there will happen unnecessary trainings. For example if n_layer = 1 and n1_units = 16, there are four combinations that train the same network:

n_layer = 1, n1_units = 16, n2_units=16

n_layer = 1, n1_units = 16, n2_units=32

n_layer = 1, n1_units = 16, n2_units=64

n_layer = 1, n1_units = 16, n2_units=128

Because the parameter n2_units is not used for a network of one layer.

Is it possible to build the network in another way, that this will not happen?

I hope my explanations are understandable :)

hyperparameters scikit-learn neural-network optimization python
1 Answers

I would pass the layers' sizes as a tuple instead of separate parameters. E.g.

def tuneModel(layer_sizes): 
    model = models.Sequential()
    model.add(keras.Input(shape=(100,)))

    for layer_size in layer_sizes:
        model.add(layers.Dense(layer_size, activation="relu"))

    model.add(layers.Dense(2))

    model.compile(loss="mse", optimizer=opt)
    return model

sizes = [16, 32, 64, 128]
shapes = (
    list(itertools.product(sizes, repeat=1))
    + list(itertools.product(sizes, repeat=2))
)

param_grid = dict(
    layer_sizes=shapes,
    batch_size=[64],
    epochs=[100]
)
user_104958930
Related