Is tensorflow row- or column-vector centric?

user_6081863 0
Viewed 507

I'm new to tensorflow, keras and a bit confused about how tf treats its input and matrix multiplication and all that jazz.

You see, in Linear Algebra (LA) you can treat contravariant vectors as columns matices(math standard)

enter image description here

or as rows matrices

enter image description here

Somewhere I've heard that:

  1. a tensor of shape (n,) e.g. [1,2,3,4,5] is not considered as a "vector" according to tf. Only tensors of shape (n,1) and (1,n) are considered vectors. But in many manuals people use those words without any system. Createing complete confusion in my head.
  2. a tensor of shape (n,1) is considered as a column vector (according to tf)
  3. but sending this column-vector (n,1) to some layer.call() as an input you can see that it is treated as a row-vector, because it's being multiplicated on the right by a self.w, but for column-oriented LA it must have been multiplicated by the self.w the left.
def call(self, inputs):
    return tf.matmul(inputs, self.w) + self.b

So the questions are these:

  1. What does it mean being x-oriented according to tf?
  2. Is tensor flow column-vector or row-vector oriented?
  3. What's up with non-vectors (n,1), and why those are not "vectors" according to TF?
  4. What is expected as an input to a layer column or row vectors?
  5. left-right matrix multiplication x*W+b in TF source code. Why it's x on the right and M on the left and not vice-versa? Why if layer expects a column-vector as an input its being multiplied by W on the right?

I see that I'm confused and can't clearly state the question. Please, be patient. Thanks.

keras tensorflow linear-algebra vector
1 Answers

It is because the elements of a rank 1 tensor are treated as a scalar.

For example

Example 1.

x = tf.constant(5)
y = tf.constant([1,2,3])

x*y will produce [5,10,15]

Example 2.

x = tf.constant([1,2,3])
y = tf.constant([[1],[2],[3]])

x*y will produce [[1 2 3] [2 4 6] [3 6 9]].

It also uses broadcast, whenever the shapes are incompatible. https://numpy.org/doc/stable/user/basics.broadcasting.html

user_196715070
Related