The model overfits the training set and fails to generalise to test set.
How could I add dropout to the feature extractor part of the model? (the .config file only provides a key value to add dropout to the box predictor)
What other measures I could take to minimize overfitting?
More details below:
I am attempting to retrain the model checkpoint "ssd_mobilenet_v1_coco_11_06_2017" on a dataset of toy animals. There are 14 classes, with 400-600 images in each. The network learns the training set in less than 30k steps. Tensorboard. The loss seems to remain quite volatile still after initial training, although I am not experienced enough to assess that.
I am testing the model by applying the exported graph to images and inspecting the results manually. (I just haven't had time to implement validation properly). The model works very well on pictures taken in very similar conditions to those in the training set. These bad test set images are randomly set aside from the training set which was acquired by taking many images in succession with slight changes in camera angle. The training set also includes various lighting conditions, backgrounds, distortions and camera locations. I would estimate it gets the classes and locations right in roughly 95% of the images from the bad test set. From this I conclude that the model fits the training set very well and can generalise a little bit.
However, the model performs very poorly on pictures taken separately with a different camera at a different time (i.e. there should be much less correlation between this test set and the training set). I would estimate the performance in this good test set is roughly 25%. From this I conclude that the model is overfitting and failing to generalise.
I have tried making some changes in the .config file.
Increasing the l2_regularizer weight from 0.00004 to 0.0001 for both the feature extractor and box predictor.
Setting box predictor
use_dropouttotruein order to enable the 20% dropout.
I am using Tensorflow 1.4 pip install and models cloned from github from about 3 weeks ago.
I call the train.py in object_detection with the following arguments:
python train.py --logtostderr --train_dir=/home/X/TrainDir/Process --pipeline_config_path=/home/X/ssd_mobilenet_v1_coco.config
My configuration file is the following:
# SSD with Mobilenet v1 configuration for MSCOCO Dataset.
# Users should configure the fine_tune_checkpoint field in the train config as
# well as the label_map_path and input_path fields in the train_input_reader and
# eval_input_reader. Search for "PATH_TO_BE_CONFIGURED" to find the fields that
# should be configured.
model {
ssd {
num_classes: 14
box_coder {
faster_rcnn_box_coder {
y_scale: 10.0
x_scale: 10.0
height_scale: 5.0
width_scale: 5.0
}
}
matcher {
argmax_matcher {
matched_threshold: 0.5
unmatched_threshold: 0.5
ignore_thresholds: false
negatives_lower_than_unmatched: true
force_match_for_each_row: true
}
}
similarity_calculator {
iou_similarity {
}
}
anchor_generator {
ssd_anchor_generator {
num_layers: 6
min_scale: 0.2
max_scale: 0.95
aspect_ratios: 1.0
aspect_ratios: 2.0
aspect_ratios: 0.5
aspect_ratios: 3.0
aspect_ratios: 0.3333
}
}
image_resizer {
fixed_shape_resizer {
height: 300
width: 300
}
}
box_predictor {
convolutional_box_predictor {
min_depth: 0
max_depth: 0
num_layers_before_predictor: 0
use_dropout: true
dropout_keep_probability: 0.8
kernel_size: 1
box_code_size: 4
apply_sigmoid_to_scores: false
conv_hyperparams {
activation: RELU_6,
regularizer {
l2_regularizer {
weight: 0.0001
}
}
initializer {
truncated_normal_initializer {
stddev: 0.03
mean: 0.0
}
}
batch_norm {
train: true,
scale: true,
center: true,
decay: 0.9997,
epsilon: 0.001,
}
}
}
}
feature_extractor {
type: 'ssd_mobilenet_v1'
min_depth: 16
depth_multiplier: 1.0
conv_hyperparams {
activation: RELU_6,
regularizer {
l2_regularizer {
weight: 0.0001
}
}
initializer {
truncated_normal_initializer {
stddev: 0.03
mean: 0.0
}
}
batch_norm {
train: true,
scale: true,
center: true,
decay: 0.9997,
epsilon: 0.001,
}
}
}
loss {
classification_loss {
weighted_sigmoid {
anchorwise_output: true
}
}
localization_loss {
weighted_smooth_l1 {
anchorwise_output: true
}
}
hard_example_miner {
num_hard_examples: 3000
iou_threshold: 0.99
loss_type: CLASSIFICATION
max_negatives_per_positive: 3
min_negatives_per_image: 0
}
classification_weight: 1.0
localization_weight: 1.0
}
normalize_loss_by_num_matches: true
post_processing {
batch_non_max_suppression {
score_threshold: 1e-8
iou_threshold: 0.6
max_detections_per_class: 100
max_total_detections: 100
}
score_converter: SIGMOID
}
}
}
train_config: {
batch_size: 8
optimizer {
rms_prop_optimizer: {
learning_rate: {
exponential_decay_learning_rate {
initial_learning_rate: 0.004
decay_steps: 800720
decay_factor: 0.95
}
}
momentum_optimizer_value: 0.9
decay: 0.9
epsilon: 1.0
}
}
fine_tune_checkpoint: "/home/X/tensorflow/models/research/object_detection/ssd_mobilenet_v1_coco_11_06_2017/model.ckpt"
from_detection_checkpoint: true
# Note: The below line limits the training process to 200K steps, which we
# empirically found to be sufficient enough to train the pets dataset. This
# effectively bypasses the learning rate schedule (the learning rate will
# never decay). Remove the below line to train indefinitely.
num_steps: 200000
data_augmentation_options {
random_horizontal_flip {
}
}
data_augmentation_options {
ssd_random_crop {
}
}
}
train_input_reader: {
tf_record_input_reader {
input_path: "/home/X/TrainDir/train.record"
}
label_map_path: "/home/X/TrainDir/data_label_map.pbtxt"
}
eval_config: {
num_examples: 1200
# Note: The below line limits the evaluation process to 10 evaluations.
# Remove the below line to evaluate indefinitely.
max_evals: 30
}
eval_input_reader: {
tf_record_input_reader {
input_path: "/home/X/TrainDir/test.record"
}
label_map_path: "/home/X/TrainDir/data_label_map.pbtxt"
shuffle: false
num_readers: 1
num_epochs: 1
}