Training multiple models in series in Keras for hyperparameter optimization

user_11075560 0
Viewed 873

The idea is to train multiple models using the same training dataset, changing some parameters each time in order to see which parameter works best. In order to do so I would need every model to be trained from scratch every time.

My current code (simplified) is:

scores= []

for i in range(n):
    model = Sequential()
    model.add(...)
    model.compile(...)
    model.fit(...)
    scores.append([i, model.score(...)])

for score in scores:
    print(score)

It runs as expected printing:

[0, 0.89712456798]
[1, 0.76652347349]
[2, 0.83178943210]
...

but I can't understant if the code does what described above or if, on the contrary, trains models that depend each on the previous one.

tf.keras keras neural-network python
2 Answers

Each time you call

model = Sequential()

your model is re-initialized, so the code sketch above does indeed what you want it to do, i.e. fitting a new model from scratch for each loop iteration.

user_46854710

Your code is fine. It creates each time a new model.

But I would suggest to train multiple model in prallel. I just discovered RayLib, which is well suited for this task.

https://ray.readthedocs.io/en/latest/auto_examples/plot_hyperparameter.html

Your code with 4 parallel training sessions, with 5 iterations each:

import ray

@ray.remote
def train(sequence):
    scores= []
    for i in range(sequence)
        model = Sequential()
        model.add(...)
        model.compile(...)
        model.fit(...)
        scores.append([i, model.score(...)])
    return scores

n=4
results = []
for i in range(n):
    results.append(train.remote(5))

results_data = ray.get(results)

for i in results_data:
    print(i)

You could make hyperparameter as an input in your train function and test different architectures fast. RayLib requires Ubuntu tho.

user_117022280
Related