Flutter - Draggable feedback widget moves(shifts) on click (without even dragging)

See the shift happening on click.

I haven't even started dragging, yet it concludes this as dragged / swiped right.

    Widget userCard = UserCard(user: User.users[0]);

                 body: Column(mainAxisAlignment: MainAxisAlignment.center, children: [
                  Draggable(
                    feedback:
                        userCard,
                    onDragEnd: (drag) {
                      print(drag.velocity.pixelsPerSecond.dx);
                      if (drag.velocity.pixelsPerSecond.dx < 0.0) {
                        print('Swiped Left');
                      } else {
                        print('Swiped Right');
                      }
                    },
                    child: userCard,
                    //childWhenDragging: UserCard(user: User.users[1]),
                  ),

I understood its the feedback widget that's causing the problem. But how do I correct this?