didSelectItemAt and didDeselectItemAt are not working as expected in swift 3.0

I am using didSelectItemAt and didDeselectItemAt for multiple selection of collectionViewCell. I want to select the cell and make the border blue color if it is selected and also unselect the 'selected' cell and make the border default. But my problem is that didDeselectItemAt is getting called alternately. when once i tap on any cell then didSelectItemAt is called and if i tap on any other cell then didDeselectItemAt is called. This should not happen i guess. didDeselectItemAt should be called only if i am tapping on already selected cell. Please correct me if i am going wrong. I have refered this UICollectionView - didDeselectItemAtIndexPath not called if cell is selected1 but dint work for me :(

public func collectionView(_ collectionView: UICollectionView, didSelectItemAt indexPath: IndexPath)
{
   let cell = collectionView.cellForItem(at: indexPath) as!  MomentDetailCell
   let moment = self.arrOfMoments[indexPath.row] as! MomentModel
   cell.toggleSelection(moment: moment)
   self.arrOfDeletingImgs.append(moment) 
}

public func collectionView(_ collectionView: UICollectionView, didDeselectItemAt indexPath: IndexPath)
{
   let cell : MomentDetailCell = self.collectionViewImages.cellForItem(at: indexPath) as! MomentDetailCell
   let moment = self.arrOfMoments[indexPath.row] as! MomentModel
   cell.toggleSelection(moment: moment)
   self.arrOfDeletingImgs.remove(at: (find(objecToFind: moment))!)
}

// Also this is the code i am using in the class. I have also made allowsMultipleSelection true in viewdidload

 extension MomentDetailViewController : UICollectionViewDelegateFlowLayout
{
    func collectionView(_ collectionView: UICollectionView, layout collectionViewLayout: UICollectionViewLayout, sizeForItemAt indexPath: IndexPath) -> CGSize
    {
      return CGSize(width: 75, height: 75)
    }
}

// This is my customCell code

func toggleSelection( moment : MomentModel)
{
    if (isSelected)
    {
        moment.isSelected = true
        self.layer.borderWidth = 3
        self.layer.borderColor = Constant.APP_BLUE_COLOR.cgColor

    }
    else
    {
        moment.isSelected = false
        self.layer.borderWidth = 1
        self.layer.borderColor = UIColor.red.cgColor
    }
}