Slightly different shape converges to wrong number - why?

user_45843 0
Viewed 83

I'm trying to figure out why TensorFlow is doing something surprising. I've boiled it down to a test case, attempting linear regression on a trivial problem that just adds two inputs together. The weights converge to 1.0 and the bias to 0.0 as they should.

With this version of the training outputs:

train_y = [2., 3., 4.]

the cost converges to 0.0 as it should, but with this version:

train_y = [[2.], [3.], [4.]]

the cost converges to 4.0. I wouldn't be so surprised if the second version gave an error message; what's surprising is that it silently gives a wrong answer. Why is it doing this?

Full code for the test case:

import tensorflow as tf
sess = tf.InteractiveSession()
tf.set_random_seed(1)

# Parameters
epochs = 10000
learning_rate = 0.01

# Data
train_x = [[1., 1.], [1., 2.], [2., 2.]]

# It works with this version
train_y = [2., 3., 4.]

# But converges on cost 4.0 with this version
#train_y = [[2.], [3.], [4.]]

# Number of samples
n_samples = len(train_x)

# Inputs and outputs
x = tf.placeholder(tf.float32, name='x')
y = tf.placeholder(tf.float32, name='y')

# Weights
w = tf.Variable(tf.random_normal([2]), name='weight')
b = tf.Variable(tf.random_normal([]), name='bias')

# Model
pred = tf.tensordot(x, w, 1) + b
cost = tf.reduce_sum((pred-y)**2 / n_samples)
optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)

# Train
tf.global_variables_initializer().run()
for epoch in range(epochs):
    # Print update at successive doublings of time
    if epoch&(epoch-1)==0 or epoch==epochs-1:
        print('{:6}'.format(epoch), end=' ')
        print('{:12.6f}'.format(cost.eval({x: train_x, y: train_y})), end=' ')
        print('    ['+', '.join('{:8.6f}'.format(z) for z in w.eval())+']', end=' ')
        print('{:12.6f}'.format(b.eval()))
    for (x1, y1) in zip(train_x, train_y):
        optimizer.run({x: x1, y: y1})
tensorflow
1 Answers

Why?

The issue is the cost function calculation when you feed tensors of different shapes. More specifically, it is the pred - y calculation.

To show you what went wrong in this specific example while avoiding the clutter, I will use constants with the same shapes and values you mentioned above:

y0 = tf.constant([2., 3., 4.])
y1 = tf.constant([[2.], [3.], [4.]])
pred = tf.constant([2., 3., 4.])

Now, let's see the shapes of the expressions pred - y0 and pred - y1:

res0 = pred - y0
res1 = pred - y1

print(res0.shape)
print(res1.shape)

The output is:

(3,)
(3, 3)

The (3, 3) is showing that when calculating pred - y1 of shapes (3,) and (3, 1), we had a broadcasting to shape (3, 3). This also means that the tf.reduce_sum() call summed 3x3 = 9 elements rather than only 3.

You can solve this for this case by transposing y1 to (1, 3) using tf.transpose():

res1_fixed = pred - tf.transpose(y1)
print(res1_fixed.shape)

The output is now:

(1, 3)

How to fix:

Now, back to your code... simply change the following expression:

cost = tf.reduce_sum((pred-y)**2 / n_samples)

To:

cost = tf.reduce_sum((pred-tf.transpose(y))**2 / n_samples)

And you will get the convergence to zero as expected in both cases.

user_42340990
Related