Problem Description
I have constructed a fully connected model as shown by the code below:
auto* model = new Sequential({
new Linear(3072,120),
new ReLU(120),
new Linear(120,32),
new ReLU(32),
new Linear(32,10),
new Softmax(10)
});
OptimizerInfo* info = new OPTIMIZER_SGD( /*lr=*/ 0.003);
model->construct(info);
model->randInit();
model->setLoss(crossEntropyLoss);
The model works properly as expected on the MNIST dataset (with the first Linear input set to 784), as the loss decreases as more and more batches get trained. The graph looks like this:
The model performance on MNIST
However, when I switched to the CIFAR-10 Dataset, the loss does not decrease over time and fluctuate around 2.30, with each element in the output vector approaching 0.10 . This case continued for all the following epoches:
The model performance on CIFAR-10
Since I worked a little bit on the problem and checked through several docs and papers, the performance of the layered fully connected model on CIFAR-10 should be around 0.8 accuracy with CE loss below 0.5, so something clearly went wrong here....
Things I have checked:
Both datasets (MNIST and CIFAR-10) are read into the model and normalized between [0,1] properly, and there are no misalignments between labels and corresponding training sample.
no NaN or Inf appeared anywhere in the model or the loss
Though the network is based on an engine I constructed without any external dependencies, I have compared the results of all functions (GEMM, ReLU, Softmax...) with same inputs in pytorch, and there is no algorithmic errors I found in my project.
Adjustments on hyperparameters (Lr, batch_size, change of optimizers, etc) do not make any difference in the performance of the model.
The model is running on FP32 and used CUDA for computations.
For any reasonable network configurations, there are no memory leaking or memory issues occuring.
Anyone have idea about what is happening?
My project link: https://github.com/SeanEngine/SEANN_2 Plz help :3