Is the forward definition of a model executed sequentially in PyTorch or in parallel?

user_13049379 0
Viewed 295

I wanted to know if instructions in forward definition of deep models class are executed sequentially? For example:

class Net(nn.Module):
    ...
    def forward(self,x):

        #### Group 1
        y = self.conv1(x)
        y = self.conv2(y)
        y = self.conv3(y)

        ### Group 2
        z = self.conv4(x)
        z = self.conv5(z)
        z = self.conv6(z)

        out = torch.cat((y,z),dim=1)
        return out

In this case Group1 and Group2 instructions can be parallelized. But will the forward definition understand this automatically or will they be executed sequentially? If no, then how to run them in parallel?

I am running PyTorch 1.3.1

Thankyou very much

pytorch
1 Answers

They are executed sequentially, only the calculations of the operations are parallelised. As far as I'm aware, there is no direct way to let them run in parallel by PyTorch.

I'm assuming that you are expecting a performance improvement from running them in parallel, but that would be at best minimal and at worst a lot slower, because operations like convolutions are already heavily parallelised and unless the input is extremely small, all cores will be used permanently. Running multiple convolutions in parallel would result in a lot of context switches, except if you would distribute the available cores evenly, but that wouldn't really make it any faster than doing them sequentially with all cores instead.

You can observe the same behaviour if you run two PyTorch programs at the same time, for example running the following, which has 3 relatively common convolutions and uses 224x224 images (like ImageNet), which is small compared to what other models (e.g. object detection) use:

import torch
import torch.nn as nn


class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.conv1 = nn.Conv2d(3, 32, kernel_size=3, padding=1)
        self.conv2 = nn.Conv2d(32, 64, kernel_size=3, padding=1)
        self.conv3 = nn.Conv2d(64, 128, kernel_size=3, padding=1)

    def forward(self, input):
        out = self.conv1(input)
        out = self.conv2(out)
        out = self.conv3(out)
        return out


input = torch.randn((10, 3, 224, 224))
model = Model().eval()

# Running it 100 times just to create a microbenchmark
for i in range(100):
    out = model(input)

To obtain information about context switches, /usr/bin/time can be used (not the built in time).

/usr/bin/time -v python bench.py

Single run:

Elapsed (wall clock) time (h:mm:ss or m:ss): 0:22.68
Involuntary context switches: 857

Running two instances at the same time:

Elapsed (wall clock) time (h:mm:ss or m:ss): 0:43.69
Involuntary context switches: 456753

To clarify, each of the instances took about 43 seconds, that's not the accumulated time.

user_64474910
Related