Cannot deselect pre-selected uicollectionviewcells on allowsMultipleSelection

I’m having some problems with a collectionview and the multi selection. The problem comes when I’m pre-selecting it, then I cannot manage to deselect it. I've made a sample project in order to someone can check this out. I've uploaded to my github profile: https://github.com/ahhhv/CollectionViewSelection

But I'll put some code in here in case somebody knows already the answer.

// Cell
override public var isSelected: Bool {
    didSet {
        if isSelected {
            layer.borderWidth = 1
            layer.borderColor = UIColor.systemBlue.cgColor
            titleLabel.textColor = .systemBlue
        } else {
            layer.borderWidth = 0
            titleLabel.textColor = .black
        }

        titleLabel.accessibilityLabel = self.isSelected ? "\(selectedItemText) \(isSelectedText)" : selectedItemText
        titleLabel.accessibilityTraits = UIAccessibilityTraits.button
    }
}

// VC
    public func collectionView(_ collectionView: UICollectionView, cellForItemAt indexPath: IndexPath) -> UICollectionViewCell {
    guard let item = item else { return UICollectionViewCell() }

    if let cell = collectionView.dequeueReusableCell(withReuseIdentifier: ModuleSelectorCell.reuseID, for: indexPath) as? ModuleSelectorCell {
        cell.set(item: item.items[indexPath.item])

        if item.selectionType == .exclusive {
            if let selectedItem = item.selectedItems, let first = selectedItem.first {
                if first == indexPath.item {
                    cell.isSelected = true
                }
            }
        } else {
            if let selectedItems = item.selectedItems {
                selectedItems.forEach { num in
                    if num == indexPath.item {
                        cell.isSelected = true
                    }
                }
            }
        }
        
        return cell
    }

    return UICollectionViewCell()
}

I know it's a long shot but I've been struggling with this for quite some time.

Kindest regards,