RecyclerView inside of SwipeRefreshLayout not clickable

I have a recycler view inside of a refresh layout in my app and for some reason, only the first item is clickable if the recyclerview is at the very top, scroll offset = 0. If I set offset to 1 if the scroll offset is 0, all items are clickable, however, this makes the swipe to refresh stop working. Does anyone know why this happens?

<androidx.swiperefreshlayout.widget.SwipeRefreshLayout
    android:id="@+id/exchanges_swipe_refresh"
    android:layout_width="match_parent"
    android:layout_height="0dp"
    app:layout_constraintTop_toTopOf="parent"
    app:layout_constraintBottom_toBottomOf="parent">

    <androidx.recyclerview.widget.RecyclerView
        android:id="@+id/rv_exchanges"
        android:layout_width="match_parent"
        android:layout_height="match_parent"
        android:paddingVertical="8dp"
        android:visibility="invisible"
        android:clipToPadding="false"
        tools:listitem="@layout/exchanges_list_item"/>

</androidx.swiperefreshlayout.widget.SwipeRefreshLayout>