Should I transpose features or weights in Neural network?

user_601862 0
Viewed 1816

I am learning Neural network.

Here is the complete piece of code: https://github.com/udacity/deep-learning-v2-pytorch/blob/master/intro-to-pytorch/Part%201%20-%20Tensors%20in%20PyTorch%20(Exercises).ipynb

When I transpose features, I get the following output:

import torch
def activation(x):
    return 1/(1+torch.exp(-x))

### Generate some data
torch.manual_seed(7) # Set the random seed so things are predictable

# Features are 5 random normal variables
features = torch.randn((1, 5))
# True weights for our data, random normal variables again
weights = torch.randn_like(features)
# and a true bias term
bias = torch.randn((1, 1))

product = features.t() * weights + bias
output = activation(product.sum())

tensor(0.9897)

However, if I transpose weights, I get a different output:

weights_prime = weights.view(5,1)
prod = torch.mm(features, weights_prime) + bias
y_hat = activation(prod.sum())

tensor(0.1595)

Why does this happen?

Update

I took a look at the solution: https://github.com/udacity/deep-learning-v2-pytorch/blob/master/intro-to-pytorch/Part%201%20-%20Tensors%20in%20PyTorch%20(Solution).ipynb

And I saw this:

y = activation((features * weights).sum() + bias)

why can a matrix features(1,5) multiply another matrix weights(1,5) without transposing weights first?

Update 2

After read several posts, I realized that

matrixA * matrixB is different from torch.mm(matrixA,matrixB) and torch.matmul(matrixA,matrixB).

Could someone confirm my three understandings between?

  1. So the * means element-wise multiplication, whereas torch.mm() and torch.matmul() are matrix-wise multiplication.

  2. differences between torch.mm() and torch.matmul(): mm() is used specifically for 2 dimensions matrix, whereas matmul() can be used for more complicated cases.

  3. In Neutral Network for this Udacity coding exercise mentioned in my above link, it needs element-wise multiplication.

Update 3

Just to bring in the Video screenshot for someone who has the same confusion:

enter image description here

And here is the video link: https://www.youtube.com/watch?time_continue=98&v=6Z7WntXays8&feature=emb_logo

pytorch machine-learning neural-network python
2 Answers

This line is taking an outer product between the two vectors.

product = features.t() * weights + bias

The resulting shape is 5x5.

If you change this to a dot product, then output will match y_hat.

product = torch.mm(weights, features.t()) + bias
user_15094330

Looking at https://pytorch.org/docs/master/generated/torch.nn.Linear.html

The typical linear (fully connected) layer in torch uses input features of shape (N,∗,in_features) and weights of shape (out_features,in_features) to produce an output of shape (N,*,out_features). Here N is the batch size, and * is any number of other dimensions (may be none).

The implementation is:

output = input.matmul(weight.t())

So, the answer is that neither of your formulas is correct according to convention; the standard formula is the one above.

You may use a non-standard shape since you're implementing things from scratch; as long as it's consistent it may work, but I don't recommend it for learning. It's unclear what 1 and 5 is in your code, but presumably you want 5 input features and one output feature, with a batch size of 1 as well. In which case the standard shapes should be input = torch.randn((1, 5)) for batch size=1 and in_features=5, and weights = torch.randn((5, 1)) for in_features=5 and out_features=1.

There is no reason why weights should ever be the same shape as features; thus weights = torch.randn_like(features) doesn't make sense.

Lastly, for your actual questions:

"Should I transpose features or weights in Neural network?" - in torch convention, you should transpose weights, but use matmul with the features first. Other frameworks may have a different convention; as long as in_features dimension of the weights is multiplied by the num_features dimension of the input, it would work.

"Why does this happen?" - these are two completely different calculations; there is no reason to think they would produce the same result.

"So the * means element-wise multiplication, whereas torch.mm() and torch.matmul() are matrix-wise multiplication." - Yes; mm is matrix-matrix only, matmul is vector-matrix or matrix-matrix, including batched versions of same - check the docs for everything matmul can do (which is kinda a lot).

"differences between torch.mm() and torch.matmul(): mm() is used specifically for 2 dimensions matrix, whereas matmul() can be used for more complicated cases." - Yes; the big difference is that matmul can broadcast. Use it when you specifically intend that; use mm to prevent unintentional broadcasting.

"In Neutral Network for this Udacity coding exercise mentioned in my above link, it needs element-wise multiplication." - I doubt it; it's probably an error in the Udacity code. This bit of code weights = torch.randn_like(features) looks like an error in any case; the dimensions of weights have a meaning different from the dimensions of features.

user_18282890
Related