ViewPager2 not restoring position

I have a basic ViewPager2 in my layout.

<androidx.viewpager2.widget.ViewPager2
        android:id="@+id/submissionViewPager"
        android:layout_width="match_parent"
        android:layout_height="match_parent"
        tools:listitem="@layout/item_video" />

I setup the ViewPager by providing an adapter in onViewCreated, it's actually a ConcatAdapter but I'm not sure if that's important.

submissionViewPager.apply {
    orientation = ViewPager2.ORIENTATION_VERTICAL
    adapter = pagingAdapter.withLoadStateInitialAndFooter(
        initial = InitialLoadStateAdapter(pagingAdapter::retry),
        footer = AppendLoadStateAdapter(pagingAdapter::retry)
    )

    offscreenPageLimit = 2
}

I then subscribe to ViewModel state

viewLifecycleOwner.lifecycleScope.launchWhenStarted {
    viewModel.state.collectLatest { pagingAdapter.submitData(state.pagingData) }
}

The problem is that when I rotate the device, the pagingAdapter is recreated meaning it's empty until the state is collected. Because it's empty the ViewPager can't reset the current item in onRestorePendingState()

// Now we have an adapter, we can clamp the pending current item and set it
mCurrentItem = Math.max(0, Math.min(mPendingCurrentItem, adapter.getItemCount() - 1));

Am I really supposed to wait until my adapter is populated before setting it on the ViewPager or is this a bug with ViewPager2?