An issue has been passed to me to look into that is really stumping me. At first, it was under the generic guise of "There must be a network issue please look into this" (always seems to be blamed first), But on further digging, we're left with fewer and fewer clues. I'll try my best to explain the issue im stuck with.
The client is a custom .Net based application sending POST/GET requests to an IIS-based web server. Normally we see no issues, however in busy periods where lots of requests are being sent we get a Task Cancelled from the client due to no response from the server.
Looking at the network level, we only see TCP-FIN and TCP-RESET-FROM-SERVER between server and client. Nothing jumps out on Packetcaptures as an issue, be it captured on either end, or the firewalls in the middle.
TCP-FINs are expected behavior, TCP-RESET-FROM-SERVER is when the Web Server severs the connection due to default timeout (2mins).
The client application is meant to re-use connections during busy periods, rather than close after every request.
We initially thought that the Client could be maxing out sockets before retry causing this error, however (the app has a maximum of 40 concurrent, and the client is set to 1000, with 5 apps per client).
Using Netstat though we see that socket count is much lower than 40, quite often not exceeding 20 even when this error occurs. (that includes time_wait status), so not sure if this needs deeper diagnoses or can be "cleared" from suspected issues.
The Web-server has no log at all of the requests being sent to it, so the issue seems to lay between .Net sending and IIS receiving.
Has anyone experienced anything similar? or can advise on how to dig further?
Currently running Windows Server2016 on both client and Server, Firewalls are Palo Altos managing Routing. We're nowhere near hitting hardware Cap issues when this issue presents itself.
So we've gone through with packet captures at all points, and nothing jumps out there as being an issue. We cant track individual requests through packets as it's all encrypted (production instance).
The exact error the client gives is: [Error] [appstuff.Httpclient] RequestID: XXXX communication with web service invocation, exception on invoke with request type Post and request uri apistuff system.aggregate.exception: one or more errors occured ---> system.threading.tasks.task.canceled.exception. a task was cancelled ---end of inner exception stack trace---- at system.threading.tasks.task1.GetResultCore(boolean waitCompelteNotification) at appstuff.httpclient.invoke(http request type, string requesturi, object request parameter, string authentication) in appstuff.httpclient.cs:line 200 ----->inner exception #0 system.threading.tasks.taskcenceledexception: a task was cancelled <----
From our reading, the job to invoke the web request was canceled due to receiving no response.
Current thinking on our end is that the Code might not actually be passing this to the network, hence the no response.
Kind Regards,