What are the numbers in torch.transforms.normalize and how to select them?

user_913098 0
Viewed 8249

I am following some tutorials and I keep seeing different numbers that seem quite arbitrary to me in the transforms section

namely,

transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (0.5,))])

or

transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))])

or

transform = transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])

or others.

I wonder where these numbers arise, and how to know to select the correct ones?

I am about to use MNIST for sanity, but very soon to use my own unique dataset and will probably need my own normaliztion.

mnist pytorch deep-learning machine-learning python
1 Answers

Normalize in pytorch context subtracts from each instance (MNIST image in your case) the mean (the first number) and divides by the standard deviation (second number). This takes place for each channel separately, meaning in mnist you only need 2 numbers because images are grayscale, but on let's say cifar10 which has colored images you would use something along the lines of your last sform (3 numbers for mean and 3 for std).

So basically each input image in MNIST gets transformed from [0,255] to [0,1] because you transform an image to Tensor (source: https://pytorch.org/docs/stable/torchvision/transforms.html -- Converts a PIL Image or numpy.ndarray (H x W x C) in the range [0, 255] to a torch.FloatTensor of shape (C x H x W) in the range [0.0, 1.0] if the PIL Image belongs to one of the modes (L, LA, P, I, F, RGB, YCbCr, RGBA, CMYK, 1) or if the numpy.ndarray has dtype = np.uint8)

After that you want your input image to have values in a range like [0,1] or [-1,1] to help your model converge to the right direction (Many reasons why scaling takes place, e.g. NNs prefer inputs around that range to avoid gradient saturation). Now as you probably noticed passing 0.5 and 0.5 in Normalize would yield vales in range:

Min of input image = 0 -> 0-0.5 = -0.5 -> gets divided by 0.5 std -> -1

Max of input image = 255 -> toTensor -> 1 -> (1 - 0.5) / 0.5 -> 1

so it transforms your data in a range [-1, 1]

user_146137680
Related