Constant loss and accuracy in pytorch

I am training a model whose output and ground truth should be binary. It's an inception based two stream models. Inception architecture is used as an encoder and for decoder a custom based model is designed consisting of conv layers, batch normalization, up sampling and using tanh as non linearity.I have tried with relu but still no result.

Model is initializing at different values but not updating. My model's forward function is:

    def forward(self, inp):
        # Preprocessing
        out = self.conv3d_1a_7x7(inp)
        skip1 = out
        out = self.maxPool3d_2a_3x3(out)
        out = self.dropout(out)
        out = self.conv3d_2b_1x1(out)
        out = self.conv3d_2c_3x3(out)
        out = self.maxPool3d_3a_3x3(out)
        out = self.dropout(out)
        out = self.mixed_3b(out)
        skip2 = out
        out = self.mixed_3c(out)
        out = self.maxPool3d_4a_3x3(out)
        out = self.dropout(out)
        out = self.mixed_4b(out)
        
        out = self.mixed_4c(out)
        out = self.dropout(out)
        out = self.mixed_4d(out)
        skip3 = out
        out = self.dropout(out)
        out = self.mixed_4e(out)
        out = self.mixed_4f(out)
        out = self.maxPool3d_5a_2x2(out)
        out = self.dropout(out)
        out = self.mixed_5b(out)
        out = self.mixed_5c(out)
        out = self.dropout(out)

        out = self.tconv6(out, skip1,skip2,skip3)
 
        out = self.sigmoid(out)
        print("Before permutation", out.shape)
        out = out.permute(0,1,3,4,2)
        out_logits = out
        return out, out_logits

My train function is:

misc,out_logits[stream] = models[stream](data[stream])
out_softmax = torch.nn.functional.softmax(out_logits[stream], 1).requires_grad_()
val, preds = torch.max(out_logits[stream].data, 1)
preds = preds.to(device, dtype=torch.float)
gt = torch.round(gt)
gt_avg = torch.mean(gt)
gt[gt>gt_avg] = 1
gt[gt<=gt_avg] = 0
out_logits[stream] = out_logits[stream].squeeze(1)
losses[stream] = criterion(preds.cpu(), gt.cpu()).requires_grad_()
if phase == 'train':
   optimizers[stream].zero_grad()
   losses[stream].backward(retain_graph=True)
   optimizers[stream].step()
running_losses[stream] += losses[stream].item() * data[stream].shape[0]
running_corrects[stream] += torch.sum(val.cpu() == gt_c.data.cpu()).item()
correct_t = torch.sum(preds==gt_c).item()
total_t = gt_c.shape[0]*gt_c.shape[1]*gt_c.shape[2]*gt_c.shape[3]
acc_epc = 100*correct_t/total_t
for scheduler in schedulers.values():
    scheduler.step()

My loss and accuracy is always constant shown here

I have tried using different optimizers like SGD, Adam , RMSprop. Furthermore, I have tried tuning the hyperparameters but model is not converging. What am I missing?