WooCommerce: Old deleted item comes back when trying to change the quantity of other item in cart

I have created a custom popup cart using WooCommerce. It has functionalities like

• Decrease a product
• Increase a product
• Remove a product

When I remove the product from the top it goes away. I am using this logic to do that using AJAX.

public function remove_item($key)
{
    $this->refresh();
    $remove = $this->woocommerce->cart->remove_cart_item($key);
    $this->woocommerce->cart->calculate_totals();
    if ($remove) {
        $this->refresh();
        return true;
    } else {
        return false;
    }
}

Now I decrease the remaining item, from quantity 2 to 1. What I get back is the already deleted comes back with 1 quantity and the present item is set to 1.

public function set_item_quantity($key, $quantity)
{
    if (!is_numeric($quantity)) {
        return;
    }

    $job = $this->woocommerce->cart->set_quantity($key, $quantity, true);
    $this->woocommerce->cart->calculate_totals();

    if ($job) {
        $this->refresh();
        return true;
    } else {
        return false;
    }
}

This is what refresh method looks like.

public function refresh()
{
    $woocommerce = \WC();
    $this->cart = $woocommerce->cart;
    $this->woocommerce = $woocommerce;
    $this->woocommerce->cart->calculate_totals();
}

This isn't a cache issue, as I have removed my cache plugin and it still gives me issues. The same doesn't happen when the cart has just one product.

Any help is appreciated. Thanks!