InfogainLoss layer blobs failure

I am trying to build up caffe's infogain loss layer to work. I have seen the posts, solutions but for me it's still doesn't work

My data lmdb dimensions are Nx1xHxW (grayscale images) and my target image lmdb dimensions are Nx3xH/8xW/8 (rgb images). My last convolutional layer's dimension is 1x3x20x80. The output_size is 3, so I have 3 classes as my label numbers are (0,1,2) in the target lmdb image dataset.

I want to try out infogain loss layer, because I think I have class imbalance problem. Most of my images contains too much background.

After My last convolutional layer (conv3) I have these:

 layer {
    name: "loss"
    type: "SoftmaxWithLoss"
    bottom: "conv3"
    top: "loss"
 } 


 layer {
    bottom: "loss"
    bottom: "label"
    top: "infoGainLoss"
    name: "infoGainLoss"
    type: "InfogainLoss"
    infogain_loss_param {
      source: "infogainH.binaryproto"
    }
 }

My infogain matrix was generated by InfogainLoss layer post (as Shai suggested) so my H matrix is 1x1x3x3 dimension (an identity matrix). So my L is 3 as I have 3 classes. When I run the prototxt file everything is fine (dimensions are ok), but after my last convolution layer (conv3 layer) I get the following error:

 I0320 14:42:16.722874  5591 net.cpp:157] Top shape: 1 3 20 80 (4800)
 I0320 14:42:16.722882  5591 net.cpp:165] Memory required for data:   2892800
 I0320 14:42:16.722892  5591 layer_factory.hpp:77] Creating layer loss
 I0320 14:42:16.722900  5591 net.cpp:106] Creating Layer loss
 I0320 14:42:16.722906  5591 net.cpp:454] loss <- conv3
 I0320 14:42:16.722913  5591 net.cpp:411] loss -> loss
 F0320 14:42:16.722928  5591 layer.hpp:374] Check failed: ExactNumBottomBlobs() == bottom.size() (2 vs. 1) SoftmaxWithLoss Layer takes 2 bottom blob(s) as input.

I double checked, every lmdb dataset filename has been set correctly. I don't know what can be the problem. Any idea?

Dear @Shai

Thank you for your answer. I did the following as you mentioned:

 layer {
    name: "prob"
    type: "Softmax"
    bottom: "conv3"
    top: "prob"
    softmax_param { axis: 1 }
 }

 layer {
    bottom: "prob"
    bottom: "label"
    top: "infoGainLoss"
    name: "infoGainLoss"
    type: "InfogainLoss"
    infogain_loss_param {
        source: "infogainH.binaryproto"
    }
 }

But I still have error:

Top shape: 1 3 20 80 (4800)
I0320 16:30:25.110862  6689 net.cpp:165] Memory required for data: 2912000
I0320 16:30:25.110867  6689 layer_factory.hpp:77] Creating layer infoGainLoss
I0320 16:30:25.110877  6689 net.cpp:106] Creating Layer infoGainLoss
I0320 16:30:25.110884  6689 net.cpp:454] infoGainLoss <- prob
I0320 16:30:25.110889  6689 net.cpp:454] infoGainLoss <- label
I0320 16:30:25.110896  6689 net.cpp:411] infoGainLoss -> infoGainLoss
F0320 16:30:25.110965  6689 infogain_loss_layer.cpp:35] Check failed: bottom[1]->height() == 1 (20 vs. 1)