Best practices to benchmark deep models on CPU (and not GPU) in PyTorch?

user_13049379 0
Viewed 2635

I am little uncertain about how to measure execution time of deep models on CPU in PyTorch ONLY FOR INFERENCE. I list here some of them but they maybe inaccurate. Please correct them if required and mention more if required. I am running on PyTorch version 1.3.1 and Intel Xeon with 64GB RAM, 3.5GHz processor and 8 cores.

  1. Should we use time.time()?

    • I know that for GPU this is a very bad idea. For GPU I do as follows
with torch.no_grad():
    wTime = 0
    start = torch.cuda.Event(enable_timing=True)
    end = torch.cuda.Event(enable_timing=True)
    out = model(input) # JUST FOR WARMUP

    start.record()
    for i in range(200):
        input = torch.rand(1,1,2048,2048).to(device)

#        beg = time.time() DO NOT USE FOR GPU

        got = net_amplifier(low,for_amplifier)

#        wTime+=time.time()-beg DO NOT USE FOR GPU
    end.record()
    torch.cuda.synchronize()

    print('execution time in MILLISECONDS: {}'.format(start.elapsed_time(end)/200))

For this code execution was done on GPU. If I have to run them on CPU what changes should be made? Will time.time() do?

  1. Should we use volatile?
    • I think the use if volatile is now discouraged after v0.3. But will it still help if I use the eval mode and no_grad()
input = Variable(torch.randn(1, 3, 227, 227), volatile=True) 
 model(input)

  1. Should the page cache be cleared?

    • One way of doing this that I know is using sudo sh -c "/bin/echo 1 > /proc/sys/vm/drop_caches"

  2. Should I remove nn.Sequential() and directly put in forward part

    • According to this link

All the methods using copy_ take some time to execute, especially on CPU this might be slow. Also the nn.Sequential() modules are slower than just executing them on the forward pass. I think this is due to some overhead that needs to be created when executing the Sequential module.

Another thing which i do not understand on the same link is

If you are running into performance issues with these small numbers, you might try to use torch.set_flush_denormal(True) to disable denormal floating point numbers on the CPU.

  1. Should torch.set_num_threads(int) be used? If yes can a demo code be provided?

  2. What does These context managers are thread local, so they won’t work if you send work to another thread using the :module:`threading` module, etc. mean as given in the documentation.

Please list any more issues for calculating execution time in CPU. Thankyou

pytorch python
1 Answers
  1. Should we use time.time()?

Yes, it's fine for CPU

  1. Should we use volatile?

As you said it's deprecated. Since 0.4.0 torch.Tensor was merged with torch.Variable (it's deprecated as well) and torch.no_grad context manager should be used.

  1. Should the page cache be cleared?

I don't think so unless you know it's a problem

  1. Should I remove nn.Sequential() and directly put in forward part

No, torch.nn.Sequential should have no or negligible performance burden on your model. It's forward is only:

def forward(self, input):
    for module in self:
        input = module(input)
    return input

If you are running into performance issues with these small numbers, you might try to use torch.set_flush_denormal(True) to disable denormal floating point numbers on the CPU.

Flushing denormal numbers (numbers which underflow) means replacing them strictly by 0.0 which might help with your performance if you have a lot of really small numbers. Example given by PyTorch docs:

>>> torch.set_flush_denormal(True)
True
>>> torch.tensor([1e-323], dtype=torch.float64)
tensor([ 0.], dtype=torch.float64)
>>> torch.set_flush_denormal(False)
True
>>> torch.tensor([1e-323], dtype=torch.float64)
tensor(9.88131e-324 *
       [ 1.0000], dtype=torch.float64)

Should torch.set_num_threads(int) be used? If yes can a demo code be provided?

According to this document it might help if you don't allocate too many threads (probably at most as many as cores in your CPU so you might try 8).

So this piece at the beginning of your code might help:

torch.set_num_threads(8)

You may want to check numbers out and see whether and how much each value helps.

What does These context managers are thread local, so they won’t work if you send work to another thread using the :module:threading module, etc. mean as given in the documentation.

If you use module like torch.multiprocessing and run torch.multiprocessing.spawn (or a-like) and one of your processes won't get into the context manager block the gradient won't be turned off (in case of torch.no_grad). Also if you use Python's threading only the threads where the block was run into will have gradients turned off (or on, it depends).

This code will make it clear for you:

import threading

import torch


def myfunc(i, tensor):
    if i % 2 == 0:
        with torch.no_grad():
            z = tensor * 2
    else:
        z = tensor * 2
    print(i, z.requires_grad)


if __name__ == "__main__":
    tensor = torch.randn(5, requires_grad=True)
    with torch.no_grad():
        for i in range(10):
            t = threading.Thread(target=myfunc, args=(i, tensor))
            t.start()

Which outputs (order may vary):

0 False
1 True
2 False
3 True
4 False
6 False
5 True
7 True
8 False
9 True

Also notice that torch.no_grad() in __main__ has no effect on spawned threads (neither would torch.enable_grad).

Please list any more issues for calculating execution time in CPU.

Converting to torchscript (see here) might help, building PyTorch from source targeted at your architecture and it's capabilities and tons of other things, this question is too wide.

user_108864200
Related