train model in a for loop with different batch sizes

user_4088201 0
Viewed 125

I tried for looping the batch sizes to know the effect of batch sizes in the model.

print ('train_X', train_X.shape)
print ('test_X', test_X.shape)
print ('train_Y', train_Y.shape)
print ('test_Y', test_Y.shape)

model = Sequential()
model.add(Dense(64, input_dim=size, activation='relu'))
model.add(Dense(7))


batch_sizes = [32,64,128,256,512]
for batch_size in batch_sizes:  

  model.compile(optimizer="adam", loss="mean_squared_error")

  history = model.fit(train_X,
            train_Y,
            epochs=3,
            validation_data=(test_X, test_Y),
            batch_size=batch_size,
            verbose=2)

It did work well for the first batch size, but then it failed. Please see the validation loss.

Epoch 1/3
212/212 - 8s - loss: 0.4929 - val_loss: 0.1251
Epoch 2/3
212/212 - 7s - loss: 0.0121 - val_loss: 0.3445
Epoch 3/3
212/212 - 7s - loss: 0.0114 - val_loss: 0.2329

The second loop failed with strange val_loss.

Epoch 1/3
106/106 - 6s - loss: 0.3061 - val_loss: 694288384.0000
Epoch 2/3
106/106 - 6s - loss: 0.0114 - val_loss: 694312320.0000
Epoch 3/3
106/106 - 5s - loss: 0.0108 - val_loss: 694323264.0000

Any idea to implement it correctly.

keras tensorflow deep-learning
1 Answers

In your above train setup, after the first batch training, the rest of the training would take pre-trained weights. So, you should initiate your model for each batch training to receive the batch effect on your models. Here is some pointer of your need.

Dummy Data

import tensorflow as tf 
from tensorflow.keras import Sequential 
from tensorflow.keras.layers import Dense 
import numpy as np 

# dummies 
X = np.random.uniform(0,1, (1000, 10))
y = np.random.uniform(0,1, (1000, 7))

X.shape, y.shape
((1000, 10), (1000, 7))

Model list

# first model 
def model_a():
    tf.keras.backend.clear_session()
    model = Sequential()
    model.add(Dense(64, input_dim=10, activation='relu'))
    model.add(Dense(7, activation='linear'))
    return model 

# second model etc
def model_b():
    tf.keras.backend.clear_session()
    model = Sequential()
    model.add(Dense(32, input_dim=10, activation='elu'))
    model.add(Dense(64, activation='elu'))
    model.add(Dense(7, activation='linear'))
    return model 

# get all model 
models = [model_a(), model_b()]

Training

# looping 
for i in range(len(models)):
    print("-"*50)

    for batch_size in [32,64,128,256,512]:  
        print('\nbs ', batch_size)

        model = models[i]
        model.compile('adam', loss='mse')

        history = model.fit(X, y, epochs=3,
                            validation_split=0.1,
                            batch_size=batch_size,
                            verbose=2)

--------------------------------------------------

bs  32
Epoch 1/3
29/29 - 1s - loss: 0.2733 - val_loss: 0.1663

bs  64
Epoch 1/3
15/15 - 0s - loss: 0.0931 - val_loss: 0.0919

bs  128
Epoch 1/3
8/8 - 0s - loss: 0.0895 - val_loss: 0.0893

...

--------------------------------------------------

bs  32
Epoch 1/3
29/29 - 0s - loss: 0.2007 - val_loss: 0.1080

bs  64
Epoch 1/3
15/15 - 0s - loss: 0.0907 - val_loss: 0.0883

bs  128
Epoch 1/3
8/8 - 0s - loss: 0.0876 - val_loss: 0.0857

...
user_92157800
Related