SwipeRefreshLayout refreshing when touch on end of view

i have SwipeRefreshLayout implemented in following way :

<android.support.v4.widget.SwipeRefreshLayout xmlns:android="http://schemas.android.com/apk/res/android"
    android:id="@+id/swipeRefreshLayout"
    android:layout_width="wrap_content"
    android:layout_height="200dp">

    <RelativeLayout
        android:id="@+id/profile_pic_name"
        android:layout_width="match_parent"
        android:layout_height="200dp"
        android:background="@drawable/profile_bg"
        android:gravity="center"
        android:paddingBottom="2dp"
        android:clickable="true"
        android:paddingTop="2dp">

        <ImageView
            android:id="@+id/profile"
            android:layout_width="200dp"
            android:layout_height="150dp"
            android:layout_alignParentTop="true"
            android:layout_centerHorizontal="true"
            android:paddingBottom="2dp"
            android:paddingTop="2dp"
            android:src="@drawable/user_default" />

        <TextView
            android:id="@+id/profile_name"
            android:layout_width="match_parent"
            android:layout_height="30dp"
            android:layout_alignParentLeft="true"
            android:layout_alignParentStart="true"
            android:layout_below="@+id/image_profile"
            android:gravity="center_horizontal"
            android:text="Hi Member"
            android:textAllCaps="true"
            android:textAppearance="?android:attr/textAppearanceLarge"
            android:textColor="@color/White"
            android:textSize="@dimen/main_menu_name_text_size"
            android:textStyle="bold" />
    </RelativeLayout>
</android.support.v4.widget.SwipeRefreshLayout>

In java code i just set following :

 mSwipeRefreshLayout.setOnRefreshListener(new SwipeRefreshLayout.OnRefreshListener() {
        @Override
        public void onRefresh() {
            loadData();
        }
    });

But when i touch on Ends of my View for example on profile_name or End of RelativeLayout. It refreshed the screen.

Can someone help in finding out solution why it refreshes itself without swipe??