Firebase Phone Authentication works on Simulator but not phone

I finally decided its time for testflight with my app. Authentication works perfectly in the simulator. I've tried it with test numbers (works great), I've tried it with my phone (sends verification code), and I've tried it with my parents phone (a device that has never been set up with my firebase) and it worked great.

As soon as I set it up on my device and tried to authenticate the same way, I encountered an error

here is the very simple code I took straight from firebase docs

public func startAuth(phoneNumber: String, completion: @escaping (Bool) -> Void) {
        PhoneAuthProvider.provider().verifyPhoneNumber(phoneNumber, uiDelegate: nil) { [weak self] verificationId, error in
            guard error == nil else {
                print("error verifying phone numer, Error: \n\(error!)")
                completion(false)
                return
            }
            self?.verificationId = verificationId
            completion(true)
        }
    }

When running in the simulator, it works great, when running on my phone, I get this error:

error verifying phone numer, Error:
Error Domain=FIRAuthErrorDomain Code=17999 "An internal error has occurred, print and inspect the error details for more information." UserInfo={FIRAuthErrorUserInfoNameKey=ERROR_INTERNAL_ERROR, NSLocalizedDescription=An internal error has occurred, print and inspect the error details for more information., NSUnderlyingError=0x283f43c30 {Error Domain=FIRAuthInternalErrorDomain Code=3 "(null)" UserInfo={NSUnderlyingError=0x283f9bae0 {Error Domain=com.google.HTTPStatus Code=503 "(null)" UserInfo={data={length = 207, bytes = 0x7b0a2020 22657272 6f72223a 207b0a20 ... 5d0a2020 7d0a7d0a }, data_content_type=application/json; charset=UTF-8}}, FIRAuthErrorUserInfoDeserializedResponseKey={
    code = 503;
    errors =     (
                {
            domain = global;
            message = "Error code: 33";
            reason = backendError;
        }
    );
    message = "Error code: 33";
}}}}

I've tried adding the firebase auth SDK: https://github.com/firebase/firebase-ios-sdk to the project and I get an error in a random file that says "redfinition of module 'Firebase'" and other stack overflow posts say to just delete the SDK and it'll solve your issue.

I'm really confused, why does it only work in the simulator and not my device? Is the SDK really the solution and I need to find another way around the redefinition issue?

Thanks for your help in advance!