Why are dot products backwards in Stanford's cs231n SVM?

user_2570025 0
Viewed 112

I'm watching the Youtube videos of Stanford's cs231n, and trying to do the assignments as exercice. While doing the SVM one I ran into the following piece of code:

def svm_loss_naive(W, X, y, reg):
  """
  Structured SVM loss function, naive implementation (with loops).

  Inputs have dimension D, there are C classes, and we operate on minibatches
  of N examples.

  Inputs:
  - W: A numpy array of shape (D, C) containing weights.
  - X: A numpy array of shape (N, D) containing a minibatch of data.
  - y: A numpy array of shape (N,) containing training labels; y[i] = c means
    that X[i] has label c, where 0 <= c < C.
  - reg: (float) regularization strength

  Returns a tuple of:
  - loss as single float
  - gradient with respect to weights W; an array of same shape as W
  """
  dW = np.zeros(W.shape) # initialize the gradient as zero

  # compute the loss and the gradient
  num_classes = W.shape[1]
  num_train = X.shape[0]
  loss = 0.0
  for i in range(num_train):
    scores = X[i].dot(W) # This line
    correct_class_score = scores[y[i]]
    for j in range(num_classes):
      if j == y[i]:
        continue
      margin = scores[j] - correct_class_score + 1 # note delta = 1
      if margin > 0:
        loss += margin

Heres the line I'm having trouble with:

scores = X[i].dot(W)

This is doing the product xW, shouldn't it be Wx? by that I mean W.dot(X[i])

deep-learning svm machine-learning math python
1 Answers

Because the array shapes are (D, C) and (N, D) for W and X respectively, you can't take the dot product directly, without transposing them both first (they must be (C, D)·(D, N) for matrix multiplication.

Since X.T.dot(W.T) == W.dot(X), the implementation simply reverses the order of the dot product as opposed to taking the transform of each array. Effectively, this just comes down to a decision around how the inputs are arranged. In this case the (somewhat arbitrary) decision was made to arrange the samples and features in a more intuitive way versus having the dot product as x·W.

user_95768760
Related