Deterministic pseudorandom bytes for crypto

Yes, XORKeyStream would be fine for this, and is a good design for a CSPRNG. The entire point of a stream cipher is that it generates a stream of "effectively random" values given a seed (the key and the IV). Those stream values are then XORed with the plaintext. "Effectively random" in this context means that there is no "efficient algorithm" (one that runs in polynomial time) that can distinguish this sequence from a "truly random" sequence. And that's what you want.

There's no need to pull in ChaCha20, though. You can use a built-in cipher like AES. Any block cipher can be converted into a stream cipher using one of several modes, such as CTR, OFB, or CFB. The differences between these modes don't really matter for this problem.

// Defining some seed, split across a "key" and an "iv"
key, _ := hex.DecodeString("6368616e676520746869732070617373")
iv, _ := hex.DecodeString("0123456789abcdef0123456789abcdef")

// We can turn a block cipher into a stream cipher, and AES is handy
block, err := aes.NewCipher(key)

if err != nil {
    panic(err)
}

// Convert block cipher into a stream cipher using a streaming mode like CTR
// OFB or CFB would work, too
stream := cipher.NewCTR(block, iv)

for x := 0; x < 10; x++ {
        // Create a fixed value of the size you want
    value := []byte{0}
    
    // Transform it to a random value
    stream.XORKeyStream(value, value)

    fmt.Printf("%d\n", value)
}

Playground

There are several other approaches you can use here. You can use a secure hash like SHA-256 to hash a counter (pick a random 128-bit number and keep incrementing it, hashing each value). Or you could hash the previous result (I have heard a little bit of controversy over whether it's possible for repeated hashes to impact the security of the hash. See https://crypto.stackexchange.com/questions/19392/any-weakness-when-performing-sha-256-repeatedly and https://crypto.stackexchange.com/questions/15481/will-repeated-rounds-of-sha-512-provide-random-numbers/15488 for more.)

You can also use a block cipher to do the same thing, by encrypting a counter or the previous output. That's pretty close to what the stream cipher modes are doing. You can just do it by hand, too.

If you want to dig into this, you might search for "csprng stream cipher" at crypto.stackexchange.com. That's a better place to ask for crypto advice, but IMO this is a programming-specific question, so did belong here.

In random number generation, first, limited randomness harvested from the computer's unpredictable elements. This physical randomness then cleaned from possible biases, like hashing, then the resulting smaller true randomness is stretched into many by using a pseudorandom number generator (PRNG).

PRNG's are deterministic which means that if the initial value (the initial true randomness - the seed) is known then the rest will be known. Keep this always secret!

We are not lost, the important design goal of the PRNGs is that the outputs should not be predictable from any other output. This is a strong requirement indicating that it should be impossible to learn the internal states only by looking at the outputs.

Go's crypto/rand uses the underlying system functionalities to get the physical randomness.

On Linux and FreeBSD, Reader uses getrandom(2) if available, /dev/urandom otherwise. On OpenBSD, Reader uses getentropy(2). On other Unix-like systems, Reader reads from /dev/urandom. On Windows systems, Reader uses the CryptGenRandom API. On Wasm, Reader uses the Web Crypto API.

Then one can use possible good Deterministic RBG like the Hash-DRGB, HMAC-DRGB, and CTR-DRGB as defined in NIST 800-90

You can use the x/crypto/chacha20 to generate the long deterministic random sequence. Keep the key and nonce fixed and secret then you will have a deterministic DRGB. It is very fast, and seekable, too.