Trouble Understanding broadcasting behavior for tensors

user_14492576 0
Viewed 58

I am trying to do element-wise multiplication of two tensors of dimensions (1,5,64) and (1,5). As far as I know, in spite of their dimension mismatch, broadcasting should allow this to work. So, I use this code:

x = tf.range(0,64*5)
x = tf.reshape(x, [1,5, 64])

y = tf.range(0,5)
y = tf.reshape(y, [1, 5])

prodct = x*y

This causes this error:

InvalidArgumentError: Incompatible shapes: [1,5,64] vs. [1,5] [Op:Mul]

However If i reshape first tensor to dimension (1,64,5), then it works. Code:

x = tf.range(0,64*5)
x = tf.reshape(x, [1,64, 5])

y = tf.range(0,5)
y = tf.reshape(y, [1, 5])

prodct = x*y

I do not understand why the first code does not work.

tensor array-broadcasting tensorflow multiplication numpy
1 Answers

The General Broadcasting Rules, when operating on two arrays, numpy compares their shapes element-wise. It starts with the trailing (i.e. rightmost) dimensions and works its way left. Two dimensions are compatible when

  • they are equal, or
  • one of them is 1

If these conditions are not met, a ValueError: operands could not be broadcast together exception is thrown, indicating that the arrays have incompatible shapes. The size of the resulting array is the size that is not 1 along each axis of the inputs.

tensorflow also follows the same spirit. Check the documentation for more examples and details. For your case, the rightmost dimension doesn't follow the rules and throws an error.

1, 5, 64
   1, 5

But this would work as it obeys the rules.

1, 64, 5
   1,  5

Code

In numpy, and in tensorflow for reference.

import numpy as np 
a = np.arange(64*5).reshape(1, 64, 5)
b = np.arange(5).reshape(1,5)
(a*b).shape
(1, 64, 5)

import tensorflow as tf 
x = tf.reshape(tf.range(0,64*5), [1, 64, 5])
y = tf.reshape(tf.range(0,5), [1, 5])
(x*y).shape
TensorShape([1, 64, 5])
user_92157800
Related