How to fetch exponent and modulus in Little Endian format from a byte array of RSA public key

I have a requirement where I have a byte array in Little Endian format as input. I need to fetch exponent and modulus from it, create a public key and use it to encrypt data. Hence I put following piece of code:

    exponentLen := 4
        exponentArr := key[:exponentLen] - ///Where key is the input byte array
        modulusStrOffset := exponentLen

        n := big.Int{}
        n.SetBytes(key[modulusStrOffset:])
        eb := big.Int{}
        eb.SetBytes(exponentArr)

        ex, err := strconv.Atoi(eb.String())
        if err != nil {
                return nil, errors.Wrap(err, "GetRsaPubKey: Strconv to int")
        }

        pubKey := rsa.PublicKey{N: &n, E: int(ex)}
cipherText, err := rsa.EncryptOAEP(sha256.New(), rand.Reader, pubKey, swk, nil) /// swk is the data to be encrypted

This code successfully encrypts data without throwing any error. However it fails on decryption side. This is so because SetBytes() presumes that input byte array is in Big Endian format(our byte array is however in LE). So the modulus and exponent evaluated is wrong. Is there a way to change byte order of input array from LE to BE and then create exponents and modulus in Little Endian format and then create public key and use it for encryption.. Please note I tried using:

 var n1 big.Int
 n1.SetBits(n.Bits())

but the problem persists.