Difference between just reshaping and reshaping and getting transpose?

user_14128125 0
Viewed 502

I'm currently studying CS231 assignments and I've realized something confusing. When calculating gradients, when I first reshape x then get transpose I got the correct result.

x_r=x.reshape(x.shape[0],-1)
dw= x_r.T.dot(dout)

enter image description here

However, when I reshape directly as the X.T shape it doesn't return the correct result.

dw = x.reshape(-1,x.shape[0]).dot(dout)

enter image description here

Can someone explain the following question?

How does the order of getting elements with np.reshape() change? How reshaping (N,d1,d2..dn) shaped array into N,D array differs from getting a reshaped array of (D,N) with its transpose.

machine-learning numpy python
1 Answers

While both your approaches result in arrays of same shape, there will by a difference in the order of elements due to the way numpy reads / writes elements. By default, reshape uses a C-like index order, which means the elements are read / written with the last axis index changing fastest, back to the first axis index changing slowest (taken from the documentation).

Here is an example of what that means in practice. Let's assume the following array x:

x = np.asarray([[[1, 2], [3, 4], [5, 6]], [[7, 8], [9, 10], [11, 12]]])

print(x.shape) # (2, 3, 2)

print(x)
# output
[[[ 1  2]
  [ 3  4]
  [ 5  6]]

 [[ 7  8]
  [ 9 10]
  [11 12]]]

Now let's reshape this array the following two ways:

opt1 = x.reshape(x.shape[0], -1)
opt2 = x.reshape(-1, x.shape[0])

print(opt1.shape) # outptu: (2, 6)
print(opt2.shape) # output: (6, 2)

print(opt1)
# output:
[[ 1  2  3  4  5  6]
 [ 7  8  9 10 11 12]]

print(opt2)
# output:
[[ 1  2]
 [ 3  4]
 [ 5  6]
 [ 7  8]
 [ 9 10]
 [11 12]]

reshape first inferred the shape of the new arrays and then returned a view where it read the elements in C-like index order.

To exemplify this on opt1: since the original array x has 12 elements, it inferred that the new array opt1 must have a shape of (2, 6) (because 2*6=12). Now, reshape returns a view where:

opt1[0][0] == x[0][0][0] 
opt1[0][1] == x[0][0][1]
opt1[0][2] == x[0][1][0]
opt1[0][3] == x[0][1][1]
opt1[0][4] == x[0][2][0]
opt1[0][5] == x[0][2][1]
opt1[1][0] == x[1][0][0]
...
opt1[1][5] == x[1][2][1]

So as described above, the last axis index changes fastest and the first axis index slowest. In the same way, the output for opt2 will be computed.

You can now verify that transposing the first option will result in the same shape but a different order of elements:

opt1 = opt1.T

print(opt1.shape) # output: (6, 2)

print(opt1)
# output: 
[[ 1  7]
 [ 2  8]
 [ 3  9]
 [ 4 10]
 [ 5 11]
 [ 6 12]]

Obviously, the two approaches do not result in the same array due to element ordering, even though they will have the same shape.

user_131828850
Related