Pkcs11 ECDSA signing returning CKR_DATA_INVALID

I am using a piv on a Yubikey loaded with trusted certificates to do mutual tls with a server. I am using this golang pkcs11 library which is a wrapper over opensc-pkcs11.so

I am implementing a signer interface using the yubikey so that it can be used as a private key with go's crypo/tls library.

My signer function looks like this (I have added data in the error for debugging):

func (signer *pkcs11PrivateKeyECDSA) Sign(rand io.Reader, digest []byte, opts crypto.SignerOpts) ([]byte, error) {
    return signer.yubi.dsaGeneric(signer.handle, pkcs11.CKM_ECDSA, digest)
}

func (yubi yubiInfo) dsaGeneric(key pkcs11.ObjectHandle, mechanism uint, digest []byte) ([]byte, error) {
    var err error
    var sigBytes []byte
    var sig dsaSignature
    mech := []*pkcs11.Mechanism{pkcs11.NewMechanism(mechanism, nil)}
    fmt.Println("Siging with key: ", key)
    if err = yubi.ctx.SignInit(yubi.ses, mech, key); err != nil {
        return nil, fmt.Errorf("signer init: %v", err)
    }
    sigBytes, err = yubi.ctx.Sign(yubi.ses, digest)
    if err != nil {
        return nil, fmt.Errorf("signer sign: %v, len: %d, \n %s", err, len(digest), string(digest))
    }
    err = sig.unmarshalBytes(sigBytes)
    if err != nil {
        return nil, err
    }

    return sig.marshalDER()
}

Attempting to do a tls handshake with the signer in the above code creates the following error message:

 rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: signer sign: pkcs11: 0x20: CKR_DATA_INVALID, len: 32, \n ZX\xe0ތ\xcdў\xbf\xdeTh.\xacS\x1d\x89\xeeH\xe0\xf0$\xd1\xda\xf7\t\xfan:\xa7\b\xb6"

What might make a digest invalid to the signer? From what I found online that error means:

CKR_DATA_INVALID: The plaintext input data to a cryptographic operation is invalid. This return value has lower priority than CKR_DATA_LEN_RANGE.

The data is the right length apparently, what else is pkcs11 checking in the digest before it signs?

Thanks!