Tensorflow Object Detection API: Corrupted training images in TensorBoard

Viewed 625

Im using TensorFlow Object Detection API with TensorFlow 2 and I have a problem with the training images that are displayed in TensorBoard like this:

enter image description here

even though the evaluation images look normally, so the problem is not in the data serialization.

enter image description here

To make sure that it is not done by image augmentation, I let only horizontal flip in the config file. See the full config file:

model {
  center_net {
    num_classes: 3
    feature_extractor {
      type: "resnet_v1_50_fpn"
    }
    image_resizer {
      keep_aspect_ratio_resizer {
        min_dimension: 512
        max_dimension: 512
        pad_to_max_dimension: true
      }
    }
    object_detection_task {
      task_loss_weight: 1.0
      offset_loss_weight: 1.0
      scale_loss_weight: 0.1
      localization_loss {
        l1_localization_loss {
        }
      }
    }
    object_center_params {
      object_center_loss_weight: 1.0
      min_box_overlap_iou: 0.7
      max_box_predictions: 100
      classification_loss {
        penalty_reduced_logistic_focal_loss {
          alpha: 2.0
          beta: 4.0
        }
      }
    }
  }
}

train_config: {

  batch_size: 5
  num_steps: 10000

  data_augmentation_options {
    random_horizontal_flip {
    }
  }
  optimizer {
    adam_optimizer: {
      epsilon: 1e-7  # Match tf.keras.optimizers.Adam's default.
      learning_rate: {
        cosine_decay_learning_rate {
          learning_rate_base: 0.3e-3
          total_steps: 10000
          warmup_learning_rate: 0.3e-3
          warmup_steps: 1000
        }
      }
    }
    use_moving_average: false
  }
  max_number_of_boxes: 100
  unpad_groundtruth_tensors: false

  fine_tune_checkpoint_version: V2
  fine_tune_checkpoint: "C:/ObjectDetection/FaceMaskDetection/Zoo/centernet_resnet50_v1_fpn_512x512_coco17_tpu-8/checkpoint/ckpt-0"
  fine_tune_checkpoint_type: "fine_tune"
}

train_input_reader: {
  label_map_path: "C:/ObjectDetection/FaceMaskDetection/Dataset/TFRecord/label_map.pbtxt"
  tf_record_input_reader {
    input_path: "C:/ObjectDetection/FaceMaskDetection/Dataset/TFRecord/train.record"
  }
}

eval_config: {
  metrics_set: "coco_detection_metrics"
  use_moving_averages: false
  batch_size: 1;
}

eval_input_reader: {
  label_map_path: "C:/ObjectDetection/FaceMaskDetection/Dataset/TFRecord/label_map.pbtxt"
  shuffle: false
  num_epochs: 1
  tf_record_input_reader {
    input_path: "C:/ObjectDetection/FaceMaskDetection/Dataset/TFRecord/eval.record"
  }
}

Im using centernet_resnet50_v1_fpn_512x512_coco17_tpu-8. The weird thing is that both loss and mAP look reasonable and I think that I wont be able to get those numbers with such bad training images. Is it just some visualization error?

enter image description here enter image description here

1 Answers

I'll run into similar issues before, you're images are most likely fine. This seems to be a Tensorboard/Tensorflow normalization error that affects the visualization in Tensorboard. There is an extensive issue about this here: https://github.com/tensorflow/models/issues/9115

You can read the details here and overall the only downside is that the image looks weird in Tensorboard.

Related