Android Repository pattern - how to update a nested recycler view item from view model

I am using ViewModel and repository pattern to fetch the data in the list. The items are arranged as a list of rows and products. A Row class has products in it. The products can be scrolled horizontally. I am using a recycler view with linear layout manager (horizontal orientation) nested inside another recycler view (vertical orientation). Fetching the items via a ViewModel and rendering in the recycler views is pretty straightforward. The challenge is when I try to update the items (their counts) when an item is added to the cart. When the button (plus sign) is clicked, a callback is sent to the view model via a listener. The horizontal adapter sends the request back to the container (vertical) adapter, and the vertical adapter sends it back to the view model.

// The horizontal adapter
class SimpleProductAdapter(
    private val shopId: String,
    private val listener: (product: CartProduct) -> Unit
) :
    ListAdapter<CartProduct, RecyclerView.ViewHolder>(...DiffCallbackGoesHere) {

    // ... some more things here
    fun bind(item: CartProduct?) {
        view.add_to_cart_button.setOnClickListener {
            listener(item)
        }
    }

The vertical adapter has similar structure

class RowAdapter(
    private val shopId: String,
    private val listener: (product: CartProduct) -> Unit
) :
    PagedListAdapter<Row, RecyclerView.ViewHolder>(...RowDiffCallbackGoesHere) {
    
    // ... some more things here
    fun bind(item: Row?) {
        SimpleProductAdapter(shopId) { product ->
            listener(product)
        }
    }

And the main site inside the fragment where the viewmodel calls exist:

val rowAdapter = RowAdapter(args.shopId) { product->
if (actionType == ADD_TO_CART_ACTION) viewModel.buy(product)
    .observe(viewLifecycleOwner, Observer {
        view.swipe.isRefreshing = it is Resource.Loading
        // I want to update the quantity here on success result
        if(Resource is Success) {}
    })

When the result is Success, I want to update the quantity; two things are challenging here

1. I am using PagedListAdapter from the paging library which gives me a (supposedly) immutable list of products.
2. Even if I update the PagedList and issue a notifyDataSetChanged, it would be too much to do that just to change a single count over a large set of items.

I am hoping to find a way where I can easily target the specific product to update or another alternative that I keep seeing on the web is to build a custom layout manager so that I can have a single adapter that draws everything on a single pass without having to nest the recycler views. That way updating the item would become easier (couldn't find a code example on this).

Any suggestions please.