I wrote my own autograd using numpy, and I heard pytorch prefers float32 because it's faster and saves memory. so I used float32 in numpy in calculation, and saves in float32 as well.
However, when I compares the result with pytorch, it varies by a bit (for example, the final loss comparison is: numpy autograd:0.005946858786046505 vs pytorch autograd: 0.005946869496256113). When I used float64 in numpy, the final result became similar (numpy autograd: 0.9532327802786481 vs pytorch autograd: 0.9532327802786484)
So I suspect pytorch calculates everything in float64 but saves its parameters using float32. Is it true? I can't seem to find an answer elsewhere. Thanks!