Infinite Loop with ImageDataGenerator

Question

I am working with a larger dataset and as a result, have to load data into my RAM batchwise for faster running without running out of resources. I am using Image Data Generator with .flow

Using a for loop is resulting in an infinite loop that constantly generates images of the same batch size before looping over to start again. The preparator code is shown below:

train_dataset=tf.keras.preprocessing.image.ImageDataGenerator(featurewise_center=False, samplewise_center=False,
    featurewise_std_normalization=False, samplewise_std_normalization=False,
    zca_whitening=False, rotation_range=0, width_shift_range=0.0,
    height_shift_range=0.0, brightness_range=None, shear_range=0.0, zoom_range=0.0,
    channel_shift_range=0.0, cval=0.0, horizontal_flip=False,
    vertical_flip=False, preprocessing_function=None,
    data_format=None, validation_split=0.0, dtype=None)
train_dataset.fit(X)

Followed by an attempted loop as shown below:

for images, y_batch in train_dataset.flow(X, y, batch_size=batch_size):
          print(np.shape(images))

The code just keeps returning arrays of the dimension:

(batch_size,img_size,img_size,3) (I need these images for bringing data into my RAM to carry out back prop). Note that I am not using anything like a model.fit and need to run these arrays through my proper code.

Not too sure how to add a stop condition

user_5668710 · Answer

That's it's point; to continue iterating forever. Keras' model.fit_gerentaor() or tf.keras's model.fit() handle terminating the training loop depending on the epochs and steps_per_epoch parameters.

If you want to manually train a model using an ImageDataGenerator(), you could roughly do the following:

epochs = 10
steps_per_epoch = len(x) // batch_size + 1  # we usually consider 1 epoch to be
                                            # the point where the model has seen
                                            # all the training samples at least once

generator = train_dataset.flow(X, y, batch_size=batch_size)

for e in range(epochs):
    for i, (images, y_batch) in enumerate(generator):
       model.train_on_batch(images, y_batch)  # train model for a single iteration
       if i >= steps_per_epoch:  # manually detect the end of the epoch
           break  
    generator.on_epoch_end()  # this shuffles the data at the end of each epoch

Infinite Loop with ImageDataGenerator

1 Answers