Progress bar not shown during training, Python

user_13519620 0
Viewed 541

I'm trying to train a 2D Unet, for the segmentation task. I execute this line of code:

model.fit(training_generator, epochs = params["nEpoches"],
                    validation_data=validation_generator, verbose = 1, use_multiprocessing = True, workers = 6, callbacks=[callbacks_list,csv_logger])

Where

  • training_generator = Istance of DataGenerator(x_training, y_train_flat, **params), with the image and the masks array as parameters of this class.
  • epochs = 2
  • validation_generator = Istance of DataGenerator(x_validation, y_validation_flat, **params), with validation data.
  • callbacks_list = checkPoint = ModelCheckpoint(filepath, monitor='val_loss', verbose=1, save_best_only=False, mode='min', period=1) callbacks_list = checkPoint

With the verbose=1 parameter I think I should see a progress bar showing the training status for each epoch, but the only thing I see is Epoch 1/2, without any bar. So I can't say if the training process is going on or if it's stucked somewhere.

keras tensorflow training-data progress python
1 Answers

According to Tensorflow documentation,

steps_per_epoch:-

Integer or None. Total number of steps (batches of samples) before declaring one epoch finished and starting the next epoch. When training with input tensors such as TensorFlow data tensors, the default None is equal to the number of samples in your dataset divided by the batch size, or 1 if that cannot be determined. If x is a tf.data dataset, and 'steps_per_epoch' is None, the epoch will run until the input dataset is exhausted. When passing an infinitely repeating dataset, you must specify the steps_per_epoch argument.

validation_steps:-

Only relevant if validation_data is provided and is a tf.data dataset. Total number of steps (batches of samples) to draw before stopping when performing validation at the end of every epoch. If 'validation_steps' is None, validation will run until the validation_data dataset is exhausted. In the case of an infinitely repeated dataset, it will run into an infinite loop. If 'validation_steps' is specified and only part of the dataset will be consumed, the evaluation will start from the beginning of the dataset at each epoch. This ensures that the same validation samples are used every time.

In your case, training progress is going on, as rightly mentioned by @Kaveh, it does not know how much steps it should have for one epoch and ran into an infinite loop. Check your batch size and add steps_per_epoch and validation_steps to the model.fit() as shown below will resolve your issue.

model.fit(training_generator, 
    steps_per_epoch = len(training_generator) // training_generator.batch_size,
    epochs = params["nEpoches"],
    validation_data=validation_generator, 
    validation_steps=len(validation_generator) // validation_generator.batch_size,
    verbose = 1, 
    use_multiprocessing = True, workers = 6, callbacks=[callbacks_list,csv_logger])

For more information you can refer here

user_142906810
Related