I get the following output when using pytorch profiler. Is the inference time of my network CPU Time + CUDA Time i.e (7.299s + 0.002721s = 7.2992721s) or is it either one of them depending on what I hardware I intend to use.
------------------------------------------------------- ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------
Name Self CPU % Self CPU CPU total % CPU total CPU time avg Self CUDA Self CUDA % CUDA total CUDA time avg # of Calls
------------------------------------------------------- ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------
model_inference 0.12% 9.055ms 100.00% 7.299s 7.299s 0.000us 0.00% 2.721ms 2.721ms 1
aten::conv2d 0.01% 763.000us 99.65% 7.273s 90.917ms 0.000us 0.00% 1.787ms 22.337us 80
aten::convolution 0.01% 751.000us 99.64% 7.273s 90.907ms 0.000us 0.00% 1.787ms 22.337us 80
aten::_convolution 0.02% 1.242ms 99.63% 7.272s 90.898ms 0.000us 0.00% 1.787ms 22.337us 80
aten::cudnn_convolution 0.07% 5.446ms 99.59% 7.269s 115.382ms 1.620ms 59.54% 1.620ms 25.714us 63
void cudnn::detail::implicit_convolve_sgemm<float, f... 0.00% 0.000us 0.00% 0.000us 0.000us 714.000us 26.24% 714.000us 19.833us 36
volta_scudnn_winograd_128x128_ldg1_ldg4_relu_tile148... 0.00% 0.000us 0.00% 0.000us 0.000us 544.000us 19.99% 544.000us 34.000us 16
aten::batch_norm 0.01% 709.000us 0.13% 9.596ms 133.278us 0.000us 0.00% 389.000us 5.403us 72
aten::_batch_norm_impl_index 0.01% 693.000us 0.12% 8.887ms 123.431us 0.000us 0.00% 389.000us 5.403us 72
aten::cudnn_batch_norm 0.05% 4.010ms 0.11% 8.194ms 113.806us 389.000us 14.30% 389.000us 5.403us 72
------------------------------------------------------- ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------
Self CPU time total: 7.299s
Self CUDA time total: 2.721ms
If any one is curious to know what the architecture looks like, its shown below
=========================================================================================================
Layer (type:depth-idx) Output Shape Param #
=========================================================================================================
Sequential -- --
├─DoubleDec: 1 -- --
│ └─Decoder: 2 -- --
│ │ └─ModuleList: 3-1 -- 1,083,646
│ │ └─ModuleList: 3-2 -- 3,318
│ └─Decoder: 2 -- --
│ │ └─ModuleList: 3-3 -- 1,083,646
│ │ └─ModuleList: 3-4 -- 2,607
├─MobileNetv2: 1-1 [1, 3, 256, 256] --
│ └─Sequential: 2-1 [1, 32, 128, 128] --
│ │ └─Conv2d: 3-5 [1, 32, 128, 128] 864
│ │ └─BatchNorm2d: 3-6 [1, 32, 128, 128] 64
│ │ └─ReLU6: 3-7 [1, 32, 128, 128] --
│ └─Sequential: 2-2 [1, 16, 128, 128] --
│ │ └─InvertedResidualBlock: 3-8 [1, 16, 128, 128] 1,984
│ └─Sequential: 2-3 [1, 24, 64, 64] --
│ │ └─InvertedResidualBlock: 3-9 [1, 24, 64, 64] 5,136
│ │ └─InvertedResidualBlock: 3-10 [1, 24, 64, 64] 8,832
│ └─Sequential: 2-4 [1, 32, 32, 32] --
│ │ └─InvertedResidualBlock: 3-11 [1, 32, 32, 32] 10,000
│ │ └─InvertedResidualBlock: 3-12 [1, 32, 32, 32] 14,848
│ │ └─InvertedResidualBlock: 3-13 [1, 32, 32, 32] 14,848
│ └─Sequential: 2-5 [1, 64, 16, 16] --
│ │ └─InvertedResidualBlock: 3-14 [1, 64, 16, 16] 21,056
│ │ └─InvertedResidualBlock: 3-15 [1, 64, 16, 16] 54,272
│ │ └─InvertedResidualBlock: 3-16 [1, 64, 16, 16] 54,272
│ │ └─InvertedResidualBlock: 3-17 [1, 64, 16, 16] 54,272
│ └─Sequential: 2-6 [1, 96, 16, 16] --
│ │ └─InvertedResidualBlock: 3-18 [1, 96, 16, 16] 66,624
│ │ └─InvertedResidualBlock: 3-19 [1, 96, 16, 16] 118,272
│ │ └─InvertedResidualBlock: 3-20 [1, 96, 16, 16] 118,272
│ └─Sequential: 2-7 [1, 160, 8, 8] --
│ │ └─InvertedResidualBlock: 3-21 [1, 160, 8, 8] 155,264
│ │ └─InvertedResidualBlock: 3-22 [1, 160, 8, 8] 320,000
│ │ └─InvertedResidualBlock: 3-23 [1, 160, 8, 8] 320,000
│ └─Sequential: 2-8 [1, 320, 8, 8] --
│ │ └─InvertedResidualBlock: 3-24 [1, 320, 8, 8] 473,920
├─DoubleDec: 1-2 [1, 42, 32, 32] --
│ └─Decoder: 2-9 [1, 42, 32, 32] --
│ └─Decoder: 2-10 [1, 33, 32, 32] --
=========================================================================================================
Total params: 3,986,017
Trainable params: 3,986,017
Non-trainable params: 0
Total mult-adds (G): 1.84
=========================================================================================================
Input size (MB): 0.79
Forward/backward pass size (MB): 255.35
Params size (MB): 15.94
Estimated Total Size (MB): 272.08