Toolbar is scrollable eventhough recyclerview is empty

I've got the following layout:

<?xml version="1.0" encoding="utf-8"?>
<androidx.coordinatorlayout.widget.CoordinatorLayout xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:app="http://schemas.android.com/apk/res-auto"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    android:fitsSystemWindows="true"
    android:focusableInTouchMode="true"

    >

    <com.google.android.material.appbar.AppBarLayout
        android:id="@+id/abLayoutDriver"
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        android:background="@color/BrightYellowCrayola">

        <androidx.appcompat.widget.Toolbar
            android:id="@+id/tbDriver"
            app:navigationIcon="@drawable/baseline_menu_24"
            android:layout_width="match_parent"
            android:layout_height="?attr/actionBarSize"
            app:layout_scrollFlags="scroll|enterAlways"
            app:popupTheme="@style/AppTheme.PopupOverlay"
            >

            <androidx.appcompat.widget.SearchView
                android:id="@+id/svDriver"
                android:layout_width="match_parent"
                android:layout_height="wrap_content"
                app:defaultQueryHint="@string/search_drivers"
                app:iconifiedByDefault="false"
                app:searchIcon="@null"
                app:queryBackground="@android:color/transparent"
                app:submitBackground="@android:color/transparent"
                android:imeOptions="flagNoExtractUi"

                />

        </androidx.appcompat.widget.Toolbar>



    </com.google.android.material.appbar.AppBarLayout>


        <androidx.recyclerview.widget.RecyclerView
            android:id="@+id/rvListDriver"
            android:layout_width="match_parent"
            android:layout_height="wrap_content"
            android:paddingStart="@dimen/zero_margin_when_normal"
            android:paddingEnd="@dimen/zero_margin_when_normal"
            app:layoutManager="androidx.recyclerview.widget.LinearLayoutManager"
            app:layout_behavior="@string/appbar_scrolling_view_behavior"
            />

    <com.google.android.material.floatingactionbutton.FloatingActionButton
        android:id="@+id/fabAddDeleteDriver"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:layout_gravity="bottom|end"
        android:layout_margin="@dimen/fab_margin"
        android:contentDescription="@string/add_driver"
        android:minHeight="48dp"
        android:src="@drawable/baseline_person_add_24"
        app:backgroundTint="@color/BrightYellowCrayola" />

</androidx.coordinatorlayout.widget.CoordinatorLayout>

In order to hide the Toolbar when I scroll the recyclerview I use app:layout_scrollFlags="scroll|enterAlways. However, this leads to following issue. The Toolbar now itself can be scrolled up and down, even though the recyclerview is empty. If I remove the property app:layout_scrollFlags="scroll|enterAlways I don't have this problem anymore, however then the toolbar won't be scrolled down anymore when I scroll down the recyclerview..

Any idea how can solve the issue, so that the scroll only works when the recyclerview is scrolled?