Training for object detection on custom dataset (images of shop fronts), for a single class (285 images in total), running locally on CPU, with 8GB RAM gets killed after a few steps.
I'm following this blog as a reference.
Here's the console log
(tensorflow) rajaram@rajaram-Lenovo-ideapad-110-15ISK:~/tensorflow/models$ python object_detection/train.py \
> --logtostderr \
> --pipeline_config_path=/home/rajaram/tensorflow/models/object_detection/models/sf_od_model/ssd_mobilenet_v1_sf_train.config \
> --train_dir=/home/rajaram/tensorflow/models/object_detection/models/sf_od_model/train
INFO:tensorflow:Summary name Learning Rate is illegal; using Learning_Rate instead.
WARNING:tensorflow:From /home/rajaram/tensorflow/models/object_detection/meta_architectures/ssd_meta_arch.py:607: all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02.
Instructions for updating:
Please use tf.global_variables instead.
INFO:tensorflow:Summary name /clone_loss is illegal; using clone_loss instead.
2017-09-26 22:15:08.121785: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-09-26 22:15:08.122313: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-09-26 22:15:08.123308: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-09-26 22:15:08.124144: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2017-09-26 22:15:08.124658: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
2017-09-26 22:15:08.953929: I tensorflow/core/common_runtime/simple_placer.cc:697] Ignoring device specification /device:GPU:0 for node 'prefetch_queue_Dequeue' because the input edge from 'prefetch_queue' is a reference connection and already has a device field set to /device:CPU:0
INFO:tensorflow:Restoring parameters from /home/rajaram/tensorflow/models/object_detection/models/sf_od_model/ssd_mobilenet_v1_coco_11_06_2017/model.ckpt
INFO:tensorflow:Starting Session.
INFO:tensorflow:Saving checkpoint to path /home/rajaram/tensorflow/models/object_detection/models/sf_od_model/train/model.ckpt
INFO:tensorflow:Starting Queues.
INFO:tensorflow:global_step/sec: 0
INFO:tensorflow:Recording summary at step 0.
INFO:tensorflow:global_step/sec: 0
INFO:tensorflow:global_step/sec: 0
INFO:tensorflow:global_step/sec: 0
INFO:tensorflow:Recording summary at step 0.
INFO:tensorflow:Recording summary at step 0.
INFO:tensorflow:Recording summary at step 0.
INFO:tensorflow:global_step/sec: 0
INFO:tensorflow:Saving checkpoint to path /home/rajaram/tensorflow/models/object_detection/models/sf_od_model/train/model.ckpt
INFO:tensorflow:global_step/sec: 0
INFO:tensorflow:Recording summary at step 1.
INFO:tensorflow:Recording summary at step 1.
INFO:tensorflow:global_step/sec: 0.00238991
INFO:tensorflow:global_step/sec: 0
INFO:tensorflow:Recording summary at step 1.
INFO:tensorflow:global_step/sec: 0
INFO:tensorflow:global step 1: loss = 14.4365 (801.196 sec/step)
INFO:tensorflow:Recording summary at step 1.
INFO:tensorflow:Recording summary at step 1.
INFO:tensorflow:global_step/sec: 0
INFO:tensorflow:Recording summary at step 1.
INFO:tensorflow:global step 2: loss = 12.9940 (173.981 sec/step)
INFO:tensorflow:Recording summary at step 2.
INFO:tensorflow:Recording summary at step 3.
INFO:tensorflow:global step 3: loss = 12.4866 (166.656 sec/step)
INFO:tensorflow:Saving checkpoint to path /home/rajaram/tensorflow/models/object_detection/models/sf_od_model/train/model.ckpt
INFO:tensorflow:Recording summary at step 3.
INFO:tensorflow:Saving checkpoint to path /home/rajaram/tensorflow/models/object_detection/models/sf_od_model/train/model.ckpt
INFO:tensorflow:global step 4: loss = 11.2386 (162.260 sec/step)
INFO:tensorflow:Recording summary at step 4.
INFO:tensorflow:Recording summary at step 4.
INFO:tensorflow:Recording summary at step 5.
INFO:tensorflow:global step 5: loss = 10.8210 (416.903 sec/step)
INFO:tensorflow:Recording summary at step 5.
Killed
(tensorflow) rajaram@rajaram-Lenovo-ideapad-110-15ISK:~/tensorflow/models$
My thoughts and questions
1) Are image sizes an issue? - Distribution of my images are as follows: <= 400x300 (5%), between 400x300 & 640x480 (22%), between 640x480 & 800x600 (63%) and > 800x600 (22%). Although images with dimensions around 400x300 would be good enough for recognition of shopboards, there's a skew towards larger resolutions in my dataset because the next step is text recognition on those boards.
- Is this thinking right?
- Should I resize the images to a smaller dimension (if yes - what dimensions are good) and re-do the annotation before starting the whole process again?
I could train the Oxford-IIIT Pet data (~7.9k images - which took ~13 hours) for 2000 steps (num_steps = 2000 in train_config section of config file), without crashing or getting killed. So, I figure a mere 285 images should be able to run on the CPU itself.
2) Is swap memory an issue? - I've also checked the other posts (increase swap space suggestion, no follow-up & another increase swap memory suggestion) which are on similar lines but since I could train the Oxford-IIIT pet dataset on my current system settings, training as little as 285 images shouldn't kill the process.
- Is my thinking correct?
- If not and this is indeed a solution, then I need pointers and clear steps to do this.
I would like to know what's going wrong and get it running locally. I hope I have given sufficient information to get help. If not, please let me know what's needed.
---------------------------
System information
- What is the top-level directory of the model you are using: tensorflow/models (haven't updated to the new folder structure yet)
- Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes - minimal changes (following Dat Trans's template for my own data set - Github)
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 16.04.3 LTS
- TensorFlow installed from (source or binary): binary (into a virtual environment)
- TensorFlow version (use command below): 1.3.0
- Bazel version (if compiling from source): NA
- CUDA/cuDNN version: NA
- GPU model and memory: NA
- Exact command to reproduce: python object_detection/train.py --logtostderr --pipeline_config_path=/home/rajaram/tensorflow/models/object_detection/models/sf_od_model/ssd_mobilenet_v1_sf_train.config --train_dir=/home/rajaram/tensorflow/models/object_detection/models/sf_od_model/train