Error message in Application Insights:
A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. (OurApiUrlAddress:443) A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
It's always a 21 seconds TCP timeout, this is a very generic error I know, but the reason for this error is not always the same, I've been reading all the threads about this. We've been investigating this problem for months with no luck, we're also in contact with Azure team.
Important: this same site written in RUBY was using this same API without any problem in the past, the API is responsive and it's called from other sites without any problem, but this specific site was migrated from RUBY to .NET and at the same time this site was hosted in AZURE, this are the 2 big changes. This just happens when the site (remember it's hosted in Azure) calls to API / services hosted in our company, this doesn't happen when site calls a service hosted somewhere else, these makes us think the problem may be related to the company infrastructure but it can't be that alone, this has to be related to .NET and AZURE someway since these APIs and services respond perfectly to calls from other sites hosted in our network and they were working fine with the ruby version of this site. These Apis and services are not throwing this error when called in-browser from outside the company network.
The services/apis are behind a firewall but ports are perfectly configured (there are not any other traffic apps nor devices at play).
This error doesn't seem to be related to port exhaustion or SNAT, since sometimes only 1 developer alone is working in the DEV environment and he gets this socket exception error.
Just to give an idea we're getting around 250 socket exceptions a day on production, and this is just a small percentage of all the calls, so there is something that, just sometimes, is making this happen.
We know about the well known HttpClient issue when multiple instances are created, so we decided to use the Singleton approach ensuring only 1 instance per API/Service, as I'll show here, this is the call that gives more socket exceptions:
In StartUp class/file:
services.AddSingleton<IUploadApi>(new UploadApi(new HttpClient() { BaseAddress = new Uri(appSettings.Endpoints.UploadServicesUrl) }));
Part of appsettings.json:
"Endpoints": {
"UploadServicesUrl": "https://ourApiUrlAddress"
},
UploadApi.cs
public interface IUploadApi
{
Task<UploadArtworkViewModel.UploadConfigurationData> GetUploadConfiguration();
}
public class UploadApi : IUploadApi
{
private readonly HttpClient httpClient;
public UploadApi(HttpClient client)
{
httpClient = client;
}
public async Task<UploadArtworkViewModel.UploadConfigurationData> GetUploadConfiguration()
{
var response = await httpClient.GetAsync("api/GetUploadConfiguration").ConfigureAwait(false);
var json = await response.Content.ReadAsStringAsync().ConfigureAwait(false);
return JsonConvert.DeserializeObject<UploadArtworkViewModel.UploadConfigurationData>(json);
}
}
Call from controller:
model.UploadConfiguration = await UploadApi.GetUploadConfiguration().ConfigureAwait(false);
Any idea on things to test or places to look are welcome, obviously I've not been able to reproduce this one. We know there's always a 21 seconds timeout, that's a TCP timeout, but that doesn't help much. Maybe for some reason the connection is dropped or Azure is having problems (sometimes) when accessing the company network. I can post more info from application insights if needed but I don't see anything special there about the error.
EDIT - More info: It happens when any API or service is called from this MVC site Controllers, so the problem appears sporadically (still like 300 times per day) when the site server tries to reach an API or service, this makes me believe it's something related to the company infraestructure, but still no idea what it could be.
