Drag UIButton without it shifting to center [Swift 3]

So I found out how to make a button draggable using the UIPanGestureRecognizer. But the only way I know how to do it is by storing and dragging the button by the center. The problem with this is if you try and drag the button from a corner, the button instantly shifts from the corner to the center. What I'm looking for is a solution that would keep my finger on a selected place while moving without instantly locking onto the center.

The code I'm currently using:

func buttonDrag(pan: UIPanGestureRecognizer) {
    print("Being Dragged")
    if pan.state == .began {
        print("panIF")
        buttonCenter = button.center // store old button center
    }else {
        print("panELSE")
        let location = pan.location(in: view) // get pan location
        button.center = location // set button to where finger is
    }
}

Thanks in advance.