When training some models on a working cuda environment, you can get the error RuntimeError: CUDA error: CUBLAS_STATUS_INVALID_VALUE when calling cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)
What does it means and how to fix it?
It may be an incomplete error reporting of a shape error:
A mismatch in dimension of a nn.Linear module and its inpput, for example x.shape == [a, b] going into a nn.Linear(c, c, bias=False) with c not matching the shape of x, will result in this error message.
See the Pytorch forum conversation.
