Deleting last row of a section causes crash

I have surfed around Stackoverflow and found similar questions but I still can't figure it out as it seems that I am doing it correctly.

So I have a table view of custom cells. Whenever I try to delete the last row in a given section, the app crashes and gives me the following error message:

The number of sections contained in the table view after the update (1) must be equal to the number of sections contained in the table view before the update (2), plus or minus the number of sections inserted or deleted (0 inserted, 0 deleted).

Here is my code:

// shifts = [[Shift]] (Shift is my class)
func numberOfSections(in tableView: UITableView) -> Int {
    return shifts.count
}

func tableView(_ tableView: UITableView, numberOfRowsInSection section: Int) -> Int {
    return shifts[section].count
}

func tableView(_ tableView: UITableView, commit editingStyle: UITableViewCellEditingStyle, forRowAt indexPath: IndexPath) {
    if editingStyle == .delete {

        // Removing the element from my list
        shifts[indexPath.section].remove(at: indexPath.row)

        // Removing empty lists if there are any
        var indexes = [Int]()
        for i in 0..<shifts.count {
            if shifts[i].count == 0 {
                indexes.append(i)
            }
        }
        var count = 0
        for number in indexes {
            shifts.remove(at: number-count)
            count += 1
        }

        // Deleting the row in the tableView
        myTableView.deleteRows(at: [indexPath], with: UITableViewRowAnimation.bottom)

    }

The other posts had the issue in their numberOfSections and numberOfRowsInSections methods, but my methods return the count of the list, and since I delete the element from the list before I delete the row, I can't figure out the problem.

EDIT Added code by Lion's advice still throwing error.

if tableView.numberOfRows(inSection: indexPath.section) > 1 {
            myTableView.deleteRows(at: [indexPath], with: UITableViewRowAnimation.bottom)
        } else {
            let indexSet = NSMutableIndexSet()
            indexSet.add(indexPath.section - 1)
            myTableView.deleteRows(at: [indexPath], with: UITableViewRowAnimation.bottom)
            myTableView.deleteSections(indexSet as IndexSet, with: UITableViewRowAnimation.bottom)
        }

EDIT 2: SOLUTION Above code was at fault, hence throwing an error.

func tableView(_ tableView: UITableView, commit editingStyle: UITableViewCellEditingStyle, forRowAt indexPath: IndexPath) {
if editingStyle == .delete {

    // Removing the element from my list
    shifts[indexPath.section].remove(at: indexPath.row)

    // Removing empty lists if there are any
    var indexes = [Int]()
    for i in 0..<shifts.count {
        if shifts[i].count == 0 {
            indexes.append(i)
        }
    }
    var count = 0
    for number in indexes {
        shifts.remove(at: number-count)
        count += 1
    }

    // Deleting the row in the tableView
    if tableView.numberOfRows(inSection: indexPath.section) > 1 {
        myTableView.deleteRows(at: [indexPath], with: UITableViewRowAnimation.bottom)
    } else {
        let indexSet = NSMutableIndexSet()
        indexSet.add(indexPath.section) // This line was wrong in above code.
        myTableView.deleteSections(indexSet as IndexSet, with: UITableViewRowAnimation.bottom)
    }

}