keras VAE with data generator

user_14992282 0
Viewed 147

I am building a VAE loading huge dataset. The input data is 3D binary voxel data having the dimension like (batch_size, 48 , 48, 48). To load the data one by one in training, I build a generator as follows

class DataGenerator(keras.utils.Sequence):

def __init__(self, x_set, y_set, batch_size):
    self.x = x_set # path for each dataset
    self.y = y_set
    self.batch_size = batch_size

def __len__(self):
    return len(self.x) #batch size = data[0]

def __getitem__(self, idx):
    batch_x = self.x[idx]
    return np.load(batch_x).astype("float32"), None

After I tried to train the model, I got the error message like:

NotImplementedError: When subclassing the `Model` class, you should implement a `call` method.

But after tried again, somehow the model is running. Could anyone help me how to solve this problem?

And another question here, because this is not the classification problem and there is no label, I need to put just some x_test dataset without y values for validation, but I am getting errors like:


~\anaconda3\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing)
    857 
    858         # Run validation.
--> 859         if validation_data and self._should_eval(epoch, validation_freq):
    860           val_x, val_y, val_sample_weight = (
    861               data_adapter.unpack_x_y_sample_weight(validation_data))

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

Please give a comment if someone has an experience to conduct the validation for this situation.

Thanks!!

autoencoder keras python
1 Answers

Call Method Error

This error:

NotImplementedError: When subclassing the `Model` class, you should implement a `call` method.

means that your Model subclass is not implementing the call method, you need it in order to use keras high order methods (like fit for example). This method is just a wrapper that is called when you threat your model in functional way model_instance(data). During the training (when using fit method) it is called at the end of the epoch to calculate accuracy / validation / test.

You can fix just by defining it into your model class:

class MyModel(keras.model.Model):
  ...

  def call(self, data, training=False):
    # your custom code when you call the model
    # or just pass, you don't need this method
    # for training
    pass

  def train_step(self, data):
    # here is where you customazie your model training
    # using backpropagation and other custom steps.
    # Implementing this method is optional
    ...

Anyway your DataGenerator code is not well formatted for python (missed tabs or spaces in copy-paste maybe). Make also sure to have the correct indentation.

Value Error

The second error is related to the fact that you are proably passing an invalid argument to the fit y. You can just pass None or just ignore the argument, but do not pass a list or you will get errors:

# DO
model.fit(x_dataset, batch_size=my_batch_size, others_parameter=...)
# or
model.fit(x_dataset, None, my_batch_size, other_parameters...)

# DONT
model.fit(x_dataset, y=[], batch_size=my_batch_size, other_parameters=...)
# or
model.fit(x_dataset, [], batch_size, other_parameters...)

The need to validate arises from the prevention of the model overfitting and for tune its hyperparameters.

Also refer to Keras documentation regarding on VAEs.

user_133324230
Related