Nested IAsyncEnumerable memory leak

Viewed 422

I have the following code that causes a memory leak. When using an await inside two nested IAsyncEnumerables.

The Func work is a function that takes an input from the input channel and returns a IAsyncEnumerable which is iterated over an put into the output channel.

If i use one of the two lines marked with // Works! code runs as expected without building up a huge heap.

Looking at the memory profiler without the mem leak, everything looks normal the DTO(TOut) object result returned by the ´work´ enumeration is only referenced by the output channel as expected keeping a max of 10 as its bounded.

When running the code with the memory leak every DTO object from the work IAsyncEnumerable is kept in memory with a reference in the root path, to the work function and a reference to some static variable Dictionary<Int32,Task> [Statis variable Task.s_currentActiveTasks] its like the tasks generated by the IAsyncEnumerator are never completed and disposed of.

public static ChannelReader<TOut> Pipe<TIn, TOut>(this ChannelReader<TIn> input, Func<TIn, IAsyncEnumerable<TOut>> work, CancellationToken cancellationToken = default)
{
        var output = Channel.CreateBounded<TOut>(10);

        _ = Task.Run(async () =>
        {
            try
            {
                await foreach (var item in input.ReadAllAsync(cancellationToken).ConfigureAwait(false))
                {                       
                    await foreach (var result in work(item))
                    {
                        // Memory leak!
                        await output.Writer.WriteAsync(result, cancellationToken).ConfigureAwait(false);

                        // Works!
                        //while (!output.Writer.TryWrite(result))
                        //    Thread.Sleep(1);
                        
                        // Works!
                        //output.Writer.WriteAsync(result, cancellationToken).AsTask().Wait(); 
                    }
                }

                output.Writer.Complete();
            }
            catch (Exception ex)
            {
                output.Writer.Complete(ex);
            }
        });

        return output;
}

Can anyone shed some insight into this problem?

If someone can't spot something obvious wrong that I've missed, I'll try and post a minimum reproducible example. I don't have alot of experience yet with IAsyncEnumerable.

0 Answers
Related