Disable toolbar scrolling

I have the current setup:

<android.support.design.widget.CoordinatorLayout
xmlns:android="http://schemas.android.com/apk/res/android"
xmlns:app="http://schemas.android.com/apk/res-auto"
android:id="@+id/coordiator_layout_in_main"
android:layout_width="match_parent"
android:layout_height="match_parent">

<android.support.design.widget.AppBarLayout
    android:layout_width="match_parent"
    android:layout_height="wrap_content">

    <android.support.v7.widget.Toolbar
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        android:minHeight="?attr/actionBarSize"
        app:layout_scrollFlags="scroll|enterAlways"/>
</android.support.design.widget.AppBarLayout>

<RelativeLayout
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    app:layout_behavior="@string/appbar_scrolling_view_behavior">

    <android.support.design.widget.NavigationView
        android:layout_width="170dp"
        android:layout_height="match_parent"
        android:fitsSystemWindows="true"/>

    <FrameLayout
        // I place a fragment containing a viewpager containing    fragments that contain a recyclerview.... 
        android:layout_width="wrap_content"
        android:layout_height="match_parent"
        android:layout_toRightOf="@+id/nav_view">

    </FrameLayout>
</RelativeLayout>

<FrameLayout
    android:id="@+id/settings_frame"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    android:visibility="invisible">
</FrameLayout>

<android.support.design.widget.FloatingActionButton
    android:layout_width="wrap_content"
    android:layout_height="wrap_content"
    android:src="@drawable/ic_refresh"
    app:layout_anchor="@id/coordiator_layout_in_main"
    app:layout_anchorGravity="bottom|right|end"   
   app:layout_behavior="com.material.widget.MyFloatingActionButtonBehavior" />

</android.support.design.widget.CoordinatorLayout>

Now everything work as expected if I scroll inside the framelayout that contains the fragments, the toolbar slides in and out as I want

Now the point is that I would like to disable the toolbar sliding in and out if I scroll the NavView which is on the side of the framelayout (and inside the relativelayout)

But no matter if I remove all scrolling behaviors the toolbar keeps on sliding in and out (only way to disable it is remove the scroll flags form the appbarlayout, but that disable all sliding in and out of the tolbar)

Please what am I missing here? Aren't the scolling behaviours supposed to pass the scroll events to the CoordinatorLayout?