the same pretrained model with same input , running multiple times gives different outputs

user_8194195 0
Viewed 46

I load a pretrained Resnet152 from torchvision. I evaluate the model multiple times with the same input image, but each time the output is different. It's very strange. Anyone knows what is the reason? My code is

from torchvision import transforms
import torch
from torchvision import models
from PIL import Image

# load a pretrained model
model = models.resnet152(pretrained=True)

# load a image and preprocess it
preprocessor = transforms.Compose([            
 transforms.Resize(256),                    
 transforms.CenterCrop(224),                
 transforms.ToTensor(),                     
 transforms.Normalize(                     
 mean=[0.485, 0.456, 0.406],                
 std=[0.229, 0.224, 0.225]                  
 )])

input_image = Image.open('lion2.jpg')
input_tensor = preprocessor(input_image)
input_batch = torch.unsqueeze(input_tensor, 0)

# run multiple times and print output 
for k in range(5):
    model.train()
    out = model(input_batch)

    model.eval()
    out2 = model(input_batch)

    print(out2[0][:10].cpu().detach())

The output are


tensor([ 0.4722, -2.1463, -0.5993, -0.3880, -2.6292,  1.9123, -1.7939, -0.3289,
        -0.3189,  0.5306])
tensor([ 0.4407, -2.0370, -0.7397, -0.4447, -2.6059,  1.9052, -1.9715, -0.6495,
        -0.5361,  0.2618])
tensor([ 0.3874, -1.9249, -0.8254, -0.5408, -2.5266,  1.8302, -2.1151, -0.8739,
        -0.7206,  0.0478])
tensor([ 0.3150, -1.8490, -0.9004, -0.6544, -2.4615,  1.7409, -2.2083, -1.0194,
        -0.8352, -0.1017])
tensor([ 0.2310, -1.7754, -0.8858, -0.7081, -2.3238,  1.5943, -2.2625, -1.1185,
        -0.9551, -0.2954])

(If I remove either model.train() or model.eval(), the output keep constant. )

pytorch
1 Answers

The model torchvision.models.resnet152 contains batch normalization layers with track_running_stats set to True. This means that whenever the model is called in training mode (i.e., when model.train() is set), the running_mean and running_var parameters of such batch normalization layers get updated to include the data of the batch passed in that call.

In your example, each time you call the model under model.train() in the loop, this causes the running_mean and running_var parameters of all the batch normalization layers to be updated. These are then freezed at the updated values when you call model.eval(), and used in the second forward pass, which causes the outputs to be different.

Since you are passing exactly the same inputs every time in the loop, it implies that the running_mean and running_var will converge to a constant after a large number of iterations. You can check that as a result the outputs in eval() mode eventually become identical.

The standard way of evaluating networks typically involves calling model.eval() once and using it over the entire test set, and so does not exhibit this apparent discrepancy.

user_59876980
Related