Tensorflow batchsize affects precision

user_6017300 0
Viewed 50

The same input, only different batch sizes.

  • Why the outputs are different?
  • How to avoid the difference or force the output to be the same?
from tensorflow import keras
from tensorflow.keras import layers

def create_net(n_input, n_hidden, activation=None):
  seq = keras.Sequential(
      [
      layers.Dense(n_hidden, input_shape=(n_input,), 
                   activation=activation,
                   ),
      layers.Dense(159*159,
                   )
      ]
  )
  return seq

seq2 = create_net(n_input=100000, n_hidden=50, activation='linear')
print(seq2.predict(np.ones((64, 100000)), batch_size=32)[0, :])
print(seq2.predict(np.ones((64, 100000)), batch_size=1)[0, :])

output:

[-0.10577075 -0.02522129  0.05591403 ...  0.07279566 -0.01813894
 -0.03258121]
[-0.10577081 -0.02522125  0.05591398 ...  0.07279569 -0.01813904
 -0.03258121]
keras tensorflow python
1 Answers

The change in result happens only when the batch size is very less i.e 1 or 2, this is because if the batch is too small, the mean and variance for any particular batch will not be the same as compared to the larger dataset.

I tested with your code with a batch size of more than 2 and it all gave the same results.

import numpy as np
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
tf.random.set_seed(42)
np.random.seed = 42

def create_net(n_input, n_hidden, activation=None):
  seq = keras.Sequential(
      [
      layers.Dense(n_hidden, input_shape=(n_input,), 
                   activation=activation,
                   ),
      layers.Dense(159*159,
                   )
      ]
  )
  return seq

seq2 = create_net(n_input=100000, n_hidden=50, activation='linear')
print(seq2.predict(np.ones((64, 100000)), batch_size=1)[0, :])

print(seq2.predict(np.ones((64, 100000)), batch_size=4)[0, :])
print(seq2.predict(np.ones((64, 100000)), batch_size=8)[0, :])
print(seq2.predict(np.ones((64, 100000)), batch_size=16)[0, :])
print(seq2.predict(np.ones((64, 100000)), batch_size=32)[0, :])
print(seq2.predict(np.ones((64, 100000)), batch_size=64)[0, :])

Result:

[-0.03781796 -0.17668417  0.12602948 ...  0.01211533  0.12969542
 -0.05006403] # Batch size 1

[-0.03781779 -0.17668422  0.12602954 ...  0.01211521  0.12969519
 -0.05006415]
[-0.03781779 -0.17668422  0.12602954 ...  0.01211521  0.12969519
 -0.05006415]
[-0.03781779 -0.17668422  0.12602954 ...  0.01211521  0.12969519
 -0.05006415]
[-0.03781779 -0.17668422  0.12602954 ...  0.01211521  0.12969519
 -0.05006415]
[-0.03781779 -0.17668422  0.12602954 ...  0.01211521  0.12969519
 -0.05006415]
user_142906970
Related