How to run NVIDIA DeepLearningExamples for Resnet50v1.5 using int8

user_14169633 0
Viewed 43

NVIDIA DeepLearningExamples provide inference benchmark tests for Resnet50v1.5 in float32 and float16 precision but not in int8, so is there anyone that has tried running these tests in int8 before? Any tips and suggestions for how to do it would be really appreciated.

Update: I am trying to test with int8, but is currently stuck at tf.layers.conv2d complaining about type mismatch since it does not accept integer types.

tensorflow python
1 Answers

For anyone who is interested, there would be some layers that required to be updated in order to accept integer types. There are some options available:

  • Write your own custom layers (e.g. tf.layers.conv2d) to accept integer inputs.
  • Use tf.cast to cast inputs and outputs between float and integer types.
  • Export the model to some formats (e.g. SavedModel) and use some conversion tool (e.g. TensorRT, TensorFlow Lite) to convert the model to int8.

Since I need the layers themselves to be optimized for int8 but do not want to spend time implementing them by myself, I have chosen the last option.

user_141696330
Related