I am trying to debug an issue with my classifier. The issue is that it always predicts the same class for a given input despite having close to an 80% accuracy.
I trained my CNN to detect the difference between 2 classes. class A has 2575 jpegs and class B has 665 jpegs.
Could this have caused my issue with my CNN always predicting the same class? Is this too much of an imbalance between the # of items in each class? In general, will my performance improve if I make the size of both classes the same(at 665 jpegs?)?