What does Tensorflow really do when the Gradient descent optimizer is applied to a "loss" placeholder that is not a number (a tensor of size 1) but rather a vector (a 1-dimensional tensor of size 2, 3, 4, or more)?
Is it like doing the descent on the sum of the components?
The answer to your second question is "no".
As for the second: just like in the one-dimensional case (e.g. y = f(x), x in R), where the direction the algorithm takes is defined by the derivative of the function with respect to its single variable, in the multidimensional case the 'overall' direction is defined by the derivative of the function with respect to each variable.
This means the size of the step you'll take in each direction will be determined by the value of the derivative of the variable corresponding to that direction.
Since there's no way to properly type math in StackOverflow, instead of messing around with it I'll suggest you take a look at this article.
