Recyclerview inside ViewPager inside scrollview not working

I have an xml layout that has following views Scrollview->RelativeLayout->Some Views + Tablayout + ViewPager->Recylerview(In Fragment of ViewPager). ViewPager has some fixed height (Keeping it "Wrap_Content" does not show it at all).Now the issue is that Recylerview never scrolls. I have tried several solutions already posted like Wraping viewpager inside "Nested Scrollview" Or Even Making Child of LinearLayout. But Nothing Worked.

Below is my xml:

<?xml version="1.0" encoding="utf-8"?>
<RelativeLayout xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:app="http://schemas.android.com/apk/res-auto"
    android:layout_width="match_parent"
    android:layout_height="match_parent">

    <android.support.v4.widget.NestedScrollView
        android:layout_width="match_parent"
        android:layout_height="match_parent"
        android:layout_above="@+id/btn_startdraw"
        android:fillViewport="true"
        android:gravity="center">

        <RelativeLayout
            android:layout_width="match_parent"
            android:layout_height="match_parent"
            android:gravity="center"
            android:padding="@dimen/_5sdp">


            <TextView
                android:id="@+id/invites_accepted"
                style="@style/bodyStyleBlue"
                android:layout_width="wrap_content"
                android:layout_height="wrap_content"
                android:layout_below="@+id/committee_slots"
                android:text="@string/invites_accepted" />

               <!-- A lot of other Textfields for data display-->

                <android.support.design.widget.TabLayout
                android:id="@+id/tabs_pendingcommittee"
                style="@style/TabStyle"
                android:layout_width="match_parent"
                android:layout_height="wrap_content"
                android:layout_below="@+id/draw_mode_val"
                app:tabMode="fixed"></android.support.design.widget.TabLayout>b


            <com.apponative.committeeapp.ui.custom.CustomViewPager
                android:id="@+id/pending_member_view"
                android:layout_width="match_parent"
                android:layout_height="@dimen/_250sdp"
                android:layout_below="@+id/tabs_pendingcommittee"
                app:layout_behavior="@string/appbar_scrolling_view_behavior" />

        </RelativeLayout>
    </android.support.v4.widget.NestedScrollView>

    <!-- Some other views on bottom-->
</RelativeLayout>

And Fragment that Viewpager is showing has following layout xml:

<?xml version="1.0" encoding="utf-8"?>
<FrameLayout xmlns:android="http://schemas.android.com/apk/res/android"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    android:background="@color/white">

    <TextView
        style="@style/bodyStyleBold1"
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        android:text="No People..." />

    <android.support.v7.widget.RecyclerView
        android:id="@+id/contact_list"
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        android:layout_below="@+id/title"
        android:background="@color/white"></android.support.v7.widget.RecyclerView>
</FrameLayout>

I have tried using nestedScrollingEnabled and also setFixedHeight properties on Recyclerview. But n Nothing worked.