HttpClient timeout using HttpCompletionOption.ResponseHeadersRead

Viewed 1232

.NET Core 3.1 Console application on Windows, I'm trying to figure out why the httpClient.Timeout does not seem to be working when getting the content after using HttpCompletionOption.ResponseHeadersRead

static async Task Main(string[] args)
{
    var httpClient = new HttpClient();

    // if using HttpCompletionOption this timeout doesn't work
    httpClient.Timeout = TimeSpan.FromSeconds(5);

    var uri = new Uri("http://brokenlinkcheckerchecker.com/files/200MB.zip");

    // will not timeout
    //using var httpResponseMessage = await httpClient.GetAsync(uri, HttpCompletionOption.ResponseHeadersRead);

    // will timeout after 5s with a TaskCanceledException
    var httpResponseMessage = await httpClient.GetAsync(uri);

    Console.WriteLine($"Status code is {httpResponseMessage.StatusCode}. Press any key to get content");
    Console.ReadLine();
    Console.WriteLine("getting content");

    var html = await httpResponseMessage.Content.ReadAsStringAsync();
    Console.WriteLine($"finished and length is {html.Length}");
}

Have also tried a CancellationToken

// will not timeout
var cts = new CancellationTokenSource(5000);
using var httpResponseMessage = await httpClient.GetAsync(uri, HttpCompletionOption.ResponseHeadersRead,
 cts.Token);

and ReadAsStreamAsync

// will not timeout
using (Stream streamToReadFrom = await httpResponseMessage.Content.ReadAsStreamAsync())
{
    string fileToWriteTo = Path.GetTempFileName();
    using (Stream streamToWriteTo = File.Open(fileToWriteTo, FileMode.Create))
    {
        await streamToReadFrom.CopyToAsync(streamToWriteTo);
    }
}

I learned about HttpCompletionOption from this great article: https://www.stevejgordon.co.uk/using-httpcompletionoption-responseheadersread-to-improve-httpclient-performance-dotnet

Update Using @StephenCleary answer below of passing the cancellationToken into the CopyToAsync method this now works as expected.

I've included the updated code below which shows copying into a MemoryStream then into a string, which I found tricky to find how to do. For my use case this is good.

string html;
await using (var streamToReadFrom = await httpResponseMessage.Content.ReadAsStreamAsync())
await using (var streamToWriteTo = new MemoryStream())
{
    await streamToReadFrom.CopyToAsync(streamToWriteTo, cts.Token);
    // careful of what encoding - read from incoming MIME
    html = Encoding.UTF8.GetString(streamToWriteTo.ToArray());
}
1 Answers

I would expect HttpClient.Timeout to only apply to the GetAsync part of the request. HttpCompletionOption.ResponseHeadersRead means "consider the Get complete when the response headers are read", so it's complete. So the problem is that it just doesn't apply to reading from the stream.

I recommend using Polly's Timeout instead of HttpClient.Timeout; Polly is a generic library that can be used to timeout any operation.

If you don't want to use Polly at this time, you can pass the CancellationToken to Stream.CopyToAsync.

Related