Sign in with apple button not firing target function

I have an app with a min target of iOS 11. However, I need to support Sign In With Apple, so I'm adding a sign in with apple button to my stack view like this:

private var appleButton: UIControl 
    // Has to be a UIControl because ASAuthorizationAppleIDButton not supported in <iOS 13

private func setupAppleSignIn() {
    guard #available(iOS 13, *) else {
        return
    }

    let button = ASAuthorizationAppleIDButton(
        authorizationButtonType: .signIn,
        authorizationButtonStyle: .white
    )
    button.cornerRadius = button.bounds.height/2
    stackView.insertArrangedSubview(button, at: 0)

    button.addTarget(self, action: #selector(handleSignInWithApple), for: .touchUpInside)
    appleButton = button
}

@objc
func handleSignInWithApple() {
    print("fired")
}

This successfully adds the button to my stackview. However, when I tap it, it doesn't fire my handleSignInWithApple function. Why?

UPDATE

It's only firing if I drag my finger a bit and release it. If I just tap, it doesn't work! Why?