Unable to read data from the transport connection partway through download of file

user_8372325 0
Viewed 230

I am running into an interesting problem when trying to use the HttpClient GetAsync function. The specified url worked up until this past weekend with no changes on our side, and everything runs fine locally, but when deployed to our test or production servers it fails midway through the download with Unable to read data from the transport connection.

Currently I retrieve the response using

response = await client.GetAsync(uri, HttpCompletionOption.ResponseHeadersRead);

and once I have the response message (since it is just a headers read) I then download it to a file on the local file system.

//Write the response to file
using (Stream streamToReadFrom = await httpResponse.Content.ReadAsStreamAsync())
{
   using (Stream streamToWriteTo = File.Open(fileToWriteTo, FileMode.Create))
   {
       await streamToReadFrom.CopyToAsync(streamToWriteTo);
   }
}

The issue is, while it is streaming the data it will randomly throw a "Unable to read data from the transport connection: The connection was closed". Since the file is being streamed I can see the contents and size of the file on the disk and it only downloads anywhere from 20-30% of the file. I am able to download the file using a Browser (Chrome, Firefox and Edge), Postman, and running the program locally on my own machine. I am also able to download it VIA Edge ON the test server to the file system which to me rules out any firewall issue.

I've tried

  • Changing HttpCompleteOption.ResponseContentRead, but it fails out with the same error since it tries to get the entire content before writing it to a file.

  • Added the following config options (Download time out is 4 hours)

    client.DefaultRequestHeaders.Add("Connection", "Keep-Alive");
client.DefaultRequestHeaders.Add("Keep-Alive", DOWNLOAD_TIMEOUT.TotalSeconds.ToString());
client.Timeout = DOWNLOAD_TIMEOUT; //Set the max download wait time for the client
System.Net.ServicePointManager.SecurityProtocol |= SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12; //Accept 1 and 1.2 TLS

When doing the file stream, I get files that are filled up to 13120KB, 13296KB,7536KB, etc. The actual size that matches what I download from Postman and Browser is 50654KB.

Not sure what causes it to fail mid download when all other methods still work (especially on my local box). I don't believe that it is a Tls/Ssl error since it IS able to connect and download part of the file, and I don't believe that it is a firewall issue on the test and production server since I am able to hit that endpoint and download it on the respective servers.

Any help would be appreciated!

EDIT: The thing that makes this unique compared to other SO questions, is that I am able to connect and download ~30% of the file before I get the "Unable to read data from the ..." error. A lot of the other questions have that when trying to connect to the url whereas I am getting it after it successfully connected and started the download of the file.

dotnet-httpclient get c#
2 Answers

If you want to be confident to cope to infrastructure issues you could give some hardened httpclient a try. Downloader looks quite promising. It manages to proceed downloads after interrupted network connections.

await DownloadBuilder.New().WithUrl(@"https://file-examples-com.github.io/uploads/2018/04/file_example_AVI_1920_2_3MG.avi").WithDirectory(@"C:\temp").Build().StartAsync();
user_12581110

Hard to say but:

  • can be some kind of antivirus activity (disable and repeat test),
  • maybe Windows get updated (server / client side)? Is it Windows ? :)
  • check certificates (expiration etc)?
  • check this behavior from other machine in network
  • disable ssl validation in .net
  • use other librabry like RestSharp and see if error still exists
  • enable network sniffing (Fiddler) ?
user_49838330
Related