When I profile my model using pytorch profiler I get two different results for inference time. Question is what exactly is my inference time? And why is the CPU Time in milliseconds when the model runs on CPU but changes to seconds when combined with CUDA?
Code for running the model only on CPU
from torch.profiler import profile, record_function, ProfilerActivity
model = torch.load("./save_resnet18/modelresnet18epoch19.pth")
model.cpu()
dummy_input = torch.randn(1, 3, 256, 256)
for _ in range(10):
_ = model(dummy_input)
with profile(activities=[ProfilerActivity.CPU], record_shapes=True) as prof:
with record_function("model_inference"):
model(dummy_input)
print(prof.key_averages().table(sort_by="cuda_time_total", row_limit=10))
Gives the following output that gives an inference time of 91.413ms
-------------------------------- ------------ ------------ ------------ ------------ ------------ ------------
Name Self CPU % Self CPU CPU total % CPU total CPU time avg # of Calls
-------------------------------- ------------ ------------ ------------ ------------ ------------ ------------
aten::zeros 0.04% 38.000us 0.05% 50.000us 50.000us 1
aten::empty 1.74% 1.589ms 1.74% 1.589ms 2.327us 683
aten::zero_ 0.00% 2.000us 0.00% 2.000us 2.000us 1
model_inference 13.26% 12.121ms 99.95% 91.363ms 91.363ms 1
aten::conv2d 0.63% 572.000us 45.49% 41.580ms 519.750us 80
aten::convolution 0.79% 726.000us 44.86% 41.008ms 512.600us 80
aten::_convolution 1.02% 934.000us 44.07% 40.282ms 503.525us 80
aten::mkldnn_convolution 40.99% 37.467ms 41.62% 38.044ms 521.151us 73
aten::as_strided_ 0.11% 97.000us 0.11% 97.000us 1.329us 73
aten::add 0.95% 872.000us 0.95% 872.000us 10.634us 82
-------------------------------- ------------ ------------ ------------ ------------ ------------ ------------
Self CPU time total: 91.413ms
But when I run the model on CUDA using the following code
from torch.profiler import profile, record_function, ProfilerActivity
model = torch.load("./save_resnet18/modelresnet18epoch19.pth")
device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)
dummy_input = torch.randn(1, 3, 256, 256).cuda()
for _ in range(10):
_ = model(dummy_input)
with profile(activities=[ProfilerActivity.CPU], record_shapes=True) as prof:
with record_function("model_inference"):
model(dummy_input)
print(prof.key_averages().table(sort_by="cuda_time_total", row_limit=10))
I get the following output with two inference times
------------------------------------------------------- ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------
Name Self CPU % Self CPU CPU total % CPU total CPU time avg Self CUDA Self CUDA % CUDA total CUDA time avg # of Calls
------------------------------------------------------- ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------
model_inference 0.12% 9.055ms 100.00% 7.299s 7.299s 0.000us 0.00% 2.721ms 2.721ms 1
aten::conv2d 0.01% 763.000us 99.65% 7.273s 90.917ms 0.000us 0.00% 1.787ms 22.337us 80
aten::convolution 0.01% 751.000us 99.64% 7.273s 90.907ms 0.000us 0.00% 1.787ms 22.337us 80
aten::_convolution 0.02% 1.242ms 99.63% 7.272s 90.898ms 0.000us 0.00% 1.787ms 22.337us 80
aten::cudnn_convolution 0.07% 5.446ms 99.59% 7.269s 115.382ms 1.620ms 59.54% 1.620ms 25.714us 63
void cudnn::detail::implicit_convolve_sgemm<float, f... 0.00% 0.000us 0.00% 0.000us 0.000us 714.000us 26.24% 714.000us 19.833us 36
volta_scudnn_winograd_128x128_ldg1_ldg4_relu_tile148... 0.00% 0.000us 0.00% 0.000us 0.000us 544.000us 19.99% 544.000us 34.000us 16
aten::batch_norm 0.01% 709.000us 0.13% 9.596ms 133.278us 0.000us 0.00% 389.000us 5.403us 72
aten::_batch_norm_impl_index 0.01% 693.000us 0.12% 8.887ms 123.431us 0.000us 0.00% 389.000us 5.403us 72
aten::cudnn_batch_norm 0.05% 4.010ms 0.11% 8.194ms 113.806us 389.000us 14.30% 389.000us 5.403us 72
------------------------------------------------------- ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------ ------------
Self CPU time total: 7.299s
Self CUDA time total: 2.721ms
What is my model's inference time ?