Python Tensorflow - Running model.fit multiple times without reinstantiating the model

user_8220110 0
Viewed 2938

Background

I am watching a popular YouTube crash course on machine learning.

At 3:35:50, he mentions that the model is likely overfit, so fits it again with less epochs.

Since he didn't reinstantiate the model, isn't this equivalent to fitting the model with that same data, thereby continuing to overtrain it?

My Question

Assume you have a model created and data ready to go.

You run:

model.fit(train_images, train_labels, epochs=10)
model.fit(train_images, train_labels, epochs=8)

Is this equivalent to running:

model.fit(train_images, train_labels, epochs=18)

Or:

model.fit(train_images, train_labels, epochs=8)

If previously fitted data is overwritten, why does running model.fit a second time begin with the accuracy of the previous model?

In multiple other questions regarding saving and training models, the accepted solutions are to load the previously trained model, and run model.fit again.

If this will overwrite the pre-existing weights, doesn't that defeat the purpose of saving the model in the first place? Wouldn't training the model for the first time on the new data be equivalent?

What is the appropriate way to train a model across multiple, similar datasets while retaining accuracy across all of the data?

keras tensorflow python
3 Answers

Since he didn't reinstantiate the model, isn't this equivalent to fitting the model with that same data, thereby continuing to overtrain it?

You are correct! In order to check which number of epochs would do better in his example, he should have compiled the network again (that is, execute the above cell again).

Just remember that in general, whenever you instantiate a model again it most likely will start with completely new weights, totally different from past weights (unless you change this manually). So even though you keep the same amount of epochs, your final accuracy can change depending on the initial weights.

Are these two commands equivalent?

model.fit(train_images, train_labels, epochs=10)
model.fit(train_images, train_labels, epochs=8)

and

model.fit(train_images, train_labels, epochs=18)

No.

In the first case, you are training your network with some weights X going through all your training set 10 times, then you update your weights for some value y. Then you will train your network again though all your training set 8 times but now you are using a network with weights X+y.

For the second case, you will train your network through all your training data 18 times with the weights X.

This is different!

user_35855760

When you run the with 

model.fit(train_images, train_labels, epochs=10)
model.fit(train_images, train_labels, epochs=8)

as you mentioned the model are not reinitialized so the model object would have the values of the previous train in it and hence continue.

Just realized that the model is being run in colab.

then when he run 

model.fit(train_images, train_labels, epochs=10)

he is training it for the first time and finds that it is overfitting, then he aims to reduce the epoch to 8 and try it.

model.fit(train_images, train_labels, epochs=8)

What he wanted to do was run for 8 epochs but since it was in the colab and the model.fit() from the first run was still in the object it was acting as if it's running for 18 epochs and over fitted

As to how to avoid over fitting, one of the methods is to use EarlyStopping and ModelCheckpoint .

user_92043190

To avoid overfitting, you can add Dropout layers, it will dropout % of connections, you just need to add it to your model between Dense layers.

from keras.layers import Dropout
Dropout(0.2)  # droping 20% only in training
user_116502540
Related