What does normalizing along any axis mean in tensorflow?

user_3656142 0
Viewed 970

I have understood what normalizing using tf.keras.utils.normalize(x_train, axis=0) and tf.keras.utils.normalize(x_train, axis=1) mean mathematically -> If our columns are features and rows are data samples, using axis=0 means normalizing every feature w.r.t the sum of that feature across all samples. Now when we have an image of say dimension 28*28, what does the normalizing axis depend on ? Each image is a data sample, so I am confused on what axis to normalize the pixel values corresponding to a single image. In almost every tutorial and example, axis=1 is taken and the normalization is done on all the images together.

In addition to the mathematical operation, I would love to understand the reasoning behind the choice of axis too.

tensorflow python
1 Answers

Channel-wise normalization can mess up the visual representation of an image and that is always harder to conduct sanity check.

Question 1: Now when we have an image of say dimension 28*28, what does the normalizing axis depend on?

If you have 28*28*1 (assuming 1 channel), you can normalize this Image based on its arrangement in the dataset. Typically value used here is 1, because the Images are commonly Stacked rather than Concatenated.

[Image 0]     Compared to [Image 0][Image 1][Image 2][Image ...]
[Image 1]
[Image 2]
[Image ...]

Meaning you can reference it like this:

Image[0] = (28*28*1) // 28*28 is the number of Columns per Row (Image)
Image[1] = (28*28*1)

Question 2: Each image is a data sample, so I am confused on what axis to normalize the pixel values corresponding to a single image. In almost every tutorial and example, axis=1 is taken and the normalization is done on all the images together.

Normalizing Data means 2 things:

  1. Putting the data on the same scale (Scaling) that improves convergence speed and accuracy. 
Scaling is commonly 0 to 1, -1 to 1, and -1 to 1 with 0 mean.
  1. Balancing the data around a point (Centering) that fights exploding and vanishing gradients, and also increasing convergence and accuracy.

As I indicated in the answer in Question 1, the value axis = 1 is due to the images being stacked.

Image[0][28*28_pixels]
Image[1][28*28_pixels]
Image[...][28*28_pixels]

Hence, when you normalize in axis = 1 (columns) you could get the right scale considering all the values in the pixel location pixel 1 is compared to pixel 1 of all the images, which is done on the WHOLE dataset so the normalization is balanced throughout the data to a certain point.

user_130718360
Related