Copy/set a single byte in a Memory<byte>

I'm starting out with System.Buffers.MemoryPool<T> and System.Memory<T> in C#, looking to reduce allocations for byte arrays.

I have a bunch of bytes and byte arrays that I need to copy to a single byte array (for use in a method that only works with byte[], not Span/Memory). I'm doing something like this:

byte aByte = 0x01;
byte[] aByteArray = { 0x02, 0x03, 0x04 };
byte[] anotherByteArray = { 0x05, 0x06, 0x07 };

using (var buffer = MemoryPool<byte>.Shared.Rent(7))
{
    Span<byte> target;

    target = buffer.Memory.Slice(0, aByteArray.Length).Span;
    aByteArray.CopyTo(target);

    target = buffer.Memory.Slice(aByteArray.Length, anotherByteArray.Length).Span;
    aByteArray.CopyTo(target);

    // How to copy a single byte?
}

So, I've figured out how to copy byte arrays to the buffer, but can't figure out how to set a single byte. I tried buffer.Memory.Span[0] = aByte, but Span has no setter.