Custom DataGenerator tensorflow error 'ValueError: Failed to find data adapter that can handle input'

user_8077563 0
Viewed 129

I have a 2-channeled imagelike file from which I am cutting patches as training/validation datasets for a convolutional autoencoder. I am using a custom data generator from TensorFlow to use different data for each batch and epoch.

Here is my CustomDataGenerator class:

class CustomDataGenerator(tf.keras.utils.Sequence):

    def __init__(self, file, sample_size, batch_size=32, width=28, height=28, resolution=(28, 28)):
        'Initialization'
        self.sample_size = sample_size
        self.batch_size = batch_size
        self.resolution = resolution
        self.width = width
        self.height = height


    def __len__(self):
        'Denotes the number of batches per epoch'
        return int(np.floor(self.sample_size / self.batch_size))

    def __getitem__(self, index):
        'Generate one batch of data'

        batch = []
        for i in range(self.batch_size):
           ....

        x = np.asarray(batch)
        x = tf.transpose(x, [0, 2, 3, 1])
        return x, x

and training code:

...
    train_gen = data_generator.CustomDataGenerator(file=file, sample_size=10000)
    val_gen = data_generator.CustomDataGenerator(file=file, sample_size=2000)
    history = autoencoder.fit(train_gen, epochs=100, validation_data=val_gen)
...

when I run the code it throws:

ValueError: Failed to find data adapter that can handle input: <class 'data_generator.CustomDataGenerator'>, <class 'NoneType'>

in model.fit line during training.

tensorflow ==2.5.0, keras ==2.4.3

autoencoder keras tensorflow
1 Answers

Your __getitem__ method must return X, Y pairs. Why you are returning X, X instead?

The train_gen then is passed to model.fit() to train. The error is due to that you are sending X as Y parameter to your model.fit()

PS: you can use on_epoch_end. This function will be called at the end of every epoch by the fit method.

user_49150080
Related