Iterate ReadOnlySequenceSegment of a ReadOnlySequence and return to ArrayPool

I am using a ReadOnlySequence<byte> to chunk together discrete ReadOnlySequenceSegment<byte>, and I have my concrete implementation of the sequence segment:

class Chunk<T> : ReadOnlySequenceSegment<T>
{
    public Chunk(ReadOnlyMemory<T> memory)
    {
        Memory = memory;
    }
    public Chunk<T> Add(ReadOnlyMemory<T> mem)
    {
        var segment = new Chunk<T>(mem)
        {
            RunningIndex = RunningIndex + Memory.Length
        };

        Next = segment;
        return segment;
    }
}

And write a number of chained chunks to the ReadOnlySequence<byte>:

var pool = ArrayPool<byte>.Shared;
var data1 = pool.Rent(2); // and memcopy in { 10, 11 }
var data2 = pool.Rent(3); // and memcopy in { 0, 1, 2}
var data3 = pool.Rent(5); // and memcopy in { 0, 100, 120, 210, 255 }
/* snip memcopys */

var startChnk = new Chunk<byte>(new ReadOnlyMemory<byte>(data1));
var currentChnk = startChnk;
currentChnk = currentChnk.Add(new ReadOnlyMemory<byte>(data2));
currentChnk  = currentChnk.Add(new ReadOnlyMemory<byte>(data3));
var seq = new ReadOnlySequence<byte>(startChnk, 0, currentChnk, currentChnk.Memory.Length);

I plan to use ArrayPool<byte>.Shared and rent and return buffers from it, and when finished I need to iterate through the chunks of the ReadOnlySequence, returning each chunk to the pool.

I loop through the below, but cant get the actual array which I can then return - I cannot call ToArray() as that will copy - I need the underlying memory but unable as these by design are read only structures wrapping the underlying array.

How can I iterate through my ReadOnlySequence so that I can dispose of the individual ReadOnlySequenceSegment back to the ArrayPool?

foreach(var chunk in seq)
{
    // I don't know how to get the array byte[] top return to the pool
    pool.Return(chunk.????);
}