I have the following code that causes a memory leak. When using an await inside two nested IAsyncEnumerables.
The Func work is a function that takes an input from the input channel and returns a IAsyncEnumerable which is iterated over an put into the output channel.
If i use one of the two lines marked with // Works! code runs as expected without building up a huge heap.
Looking at the memory profiler without the mem leak, everything looks normal the DTO(TOut) object result returned by the ´work´ enumeration is only referenced by the output channel as expected keeping a max of 10 as its bounded.
When running the code with the memory leak every DTO object from the work IAsyncEnumerable is kept in memory with a reference in the root path, to the work function and a reference to some static variable Dictionary<Int32,Task> [Statis variable Task.s_currentActiveTasks] its like the tasks generated by the IAsyncEnumerator are never completed and disposed of.
public static ChannelReader<TOut> Pipe<TIn, TOut>(this ChannelReader<TIn> input, Func<TIn, IAsyncEnumerable<TOut>> work, CancellationToken cancellationToken = default)
{
var output = Channel.CreateBounded<TOut>(10);
_ = Task.Run(async () =>
{
try
{
await foreach (var item in input.ReadAllAsync(cancellationToken).ConfigureAwait(false))
{
await foreach (var result in work(item))
{
// Memory leak!
await output.Writer.WriteAsync(result, cancellationToken).ConfigureAwait(false);
// Works!
//while (!output.Writer.TryWrite(result))
// Thread.Sleep(1);
// Works!
//output.Writer.WriteAsync(result, cancellationToken).AsTask().Wait();
}
}
output.Writer.Complete();
}
catch (Exception ex)
{
output.Writer.Complete(ex);
}
});
return output;
}
Can anyone shed some insight into this problem?
If someone can't spot something obvious wrong that I've missed, I'll try and post a minimum reproducible example. I don't have alot of experience yet with IAsyncEnumerable.