Nested RecyclerView Prefetch not working on initial scroll only

So my problem is that I have a vertical recycler view with rows of nested horizontal recyclerviews. I have prefetch enabled and initial prefetch count set to 20 (for testing) on the nested recyclerviews, however, it is not prefetching on initial scroll down due to pending updates on the inner recyclerview adapter when adding items in onBindViewHolder of the outer adapter. Here's my outer adapter code. Each "block" is a horizontal scroll view

private class BlockAdapter(val blocks: List<Block>) : RecyclerView.Adapter<VH>() {
    private val viewCreatorsByType = HashMap<Int, Block>()

    override fun getItemViewType(position: Int): Int {
        val block = blocks[position]
        val blockKey = block.viewType.ordinal
        viewCreatorsByType[blockKey] = block
        return blockKey
    }

    override fun onCreateViewHolder(parent: ViewGroup, viewType: Int): VH {
        val block = viewCreatorsByType[viewType]!!
        return VH(block.createView(parent.context, parent))
    }

    override fun onBindViewHolder(holder: VH, position: Int) {
        holder.index = position
        holder.boundPresenter = blocks[position].presenter.apply {
            attachView(holder.view)
            resume()
            present()
        }
    }

    override fun onViewRecycled(holder: VH) {
        holder.boundPresenter?.run {
            pause()
            detach()
        }
        holder.boundPresenter = null
    }

    override fun getItemCount(): Int = blocks.size
}

class VH(val view: android.view.View) : RecyclerView.ViewHolder(view) {
    var boundPresenter: Presenter? = null
    var index: Int = 0
}

When present is called in onBindViewHolder, the inner adapter is populated and cards are added to the inner horizontal recycler view adapter, but it fails to prefetch due to pending updates on the horizontal recycler view adapter. This is found in the GapWorker class ->

        void collectPrefetchPositionsFromView(RecyclerView view, boolean nested) {
        mCount = 0;
        if (mPrefetchArray != null) {
            Arrays.fill(mPrefetchArray, -1);
        }

        final RecyclerView.LayoutManager layout = view.mLayout;
        if (view.mAdapter != null
                && layout != null
                && layout.isItemPrefetchEnabled()) {
            if (nested) {
                // nested prefetch, only if no adapter updates pending. Note: we don't query
                // view.hasPendingAdapterUpdates(), as first layout may not have occurred 
                if (!view.mAdapterHelper.hasPendingUpdates()) { //<-This is where prefetch is failing initially since the horizontal list adapter is has a pending "add" update. 
                    layout.collectInitialPrefetchPositions(view.mAdapter.getItemCount(), this);
                }
            } else {
                // momentum based prefetch, only if we trust current child/adapter state
                if (!view.hasPendingAdapterUpdates()) {
                    layout.collectAdjacentPrefetchPositions(mPrefetchDx, mPrefetchDy,
                            view.mState, this);
                }
            }

            if (mCount > layout.mPrefetchMaxCountObserved) {
                layout.mPrefetchMaxCountObserved = mCount;
                layout.mPrefetchMaxObservedInInitialPrefetch = nested;
                view.mRecycler.updateViewCacheSize();
            }
        }
    }

Once it's laid out, I can scroll up and prefetch works as expected, but that doesn't help with the initial jank when scrolling down. Is there a solution around this problem?