Intermittent 503 Service Unavailable responses in azure app services

user_2725487 0
Viewed 112

I tried reading all the possible articles about this topic but haven't seen any good solutions. On Microsoft site they mention scaling up or turning on auto-heal functionality (those seem a little more like a workaround).

Our solution consists of multiple .NET APIs that communicate with each other through http calls besides using async message broker.

We were recently informed by our client that there are short periods of time where the system stops responding.

Digging into the logs we see that for short periods of time (15-30 minutes) services start returning 503 responses for a percentage of requests. Looking more closely at the system and our logging, we also see that several incoming calls from our frontend clients are getting 503 responses from our App Gateway. We see the same symptoms on multiple App Service Plans.

We haven't seen any noticeable change in traffic when those problems occurred. We also haven't seen resource (memory, cpu) exhaustion. We are running on Isolated app service plans.

This doesn't happen often (maybe every 1-2 days) but if it occurs during peak usage it can highly disrupt user experience.

I noticed that we run only 1 instance of app service plan with autoscale functionality. We tried raising the number to 2 for one of the app service plans but still apps under that service plan returned 503s.

Any ideas what could be the cause or what further investigation steps to take ?

Edit:

I was able to find the 503 responses logged in one of our app services (picture below). It mentions load-balancer being an issue and the http sub status code according to the table provided by Bryan Trach says ,,Exception in rewrite provider (likely caused by SQL)". However when I tried googling for any kinds of solutions to this I got nothing.

enter image description here

azure-app-service-plans azure-web-app-service .net
2 Answers

@cah1r We apologize you are encountering 503 errors on your App Services.

Before we dive in a little, I just want to set the expectation that the Stack Overflow community will be limited in assisting you as we do not have access to your logs and troubleshooting tools. You ultimately should consider opening a technical support ticket with Azure as they will be able to pinpoint the cause better with access to your logs and their troubleshooting tools.

In the application logs, there should not only be a http status error (503 in this case) but also a substatus column. That column should be named Http SubStatus Code. Can you see if any of the 503 errors have a substatus? If so, try to align the value to the below chart, which might give a better idea of the type of 503 error.

  <httpsubStatus> 

    <error id ="0" description="Server unavailable" /> 

    <error id ="2" description="Concurrent requests limit reached." /> 

    <error id ="3" description="ASP.NET queue full" /> 

    <error id ="4" description="FastCGI queue full" /> 

    <error id ="12" description="WebSocket request made for site with websocket disabled.(WebSocketMonitoringModule)"     /> 

    <error id ="13" description="Number of active WebSocket requests has reached the maximum concurrent WebSocket requests allowed. (WebSocketMonitoringModule)" /> 

    <!-- Request throttler events --> 

    <error id ="28" description="Front End Request throttler queue full" /> 

    <error id ="29" description="Front End Request throttler (empty site)" /> 

    <error id ="30" description="Front End Request throttler (client disconnect)" /> 

    <error id ="31" description="Front End Request throttler hot site blocked" /> 

    <!-- file provider codes--> 

    <error id ="64" description="Exception in rewrite provider (likely caused by SQL)" /> 

    <error id ="65" description="Servers unavailable (would need to contact support further for this)" /> 

    <error id ="66" description="Generic Error" /> 

    <error id ="69" description="Server Farm is Scaling" /> 

    <error id ="70" description="Site Routed To Incorrect Worker" /> 

    <error id ="71" description="Function Scale Out" /> 

    <!-- iisnode --> 

    <error id ="1000" description="ERROR_NOT_ENOUGH_QUOTA: OnExecuteRequestHandler" /> 

    <error id ="1001" description="iisnode failed to accept a request because the application is recycling" /> 

    <error id ="1002" description="iisnode failed to initiate processing of a request" /> 

    <error id ="1003" description="iisnode was unable to establish named pipe connection to the node.exe process because the named pipe server is too busy" />         

    <!-- end iisnode -->  

  </httpsubStatus>

Also, it might be outside this list. Storage failover issues, Out of Memory errors, outbound socket exhaustion are also possible hardware situations occurring. It could also be an issue within your application.

Are all your App Service Plans in the same datacenter? If so, this leads me to wonder if it could be something like a storage failover issue if they are all on the same network storage.

If the above information is not able to help you get going in the right direction, it would be best to take this into a support ticket rather than going back and forth without access to proper data/tooling. If you do need further help, please email us at azcommunity@microsoft.com with the subject at "ATTN: Bryan" and in the body of the email please include your subscription ID.

user_83668140

So we finally figured out what was wrong. Posting it here because I hope it will help someone else.

For us the issue was that some of the outgoing traffic was being blocked in Azure firewall (by default we block all unknown traffic that is not listed as needed by Microsoft). When we whitelisted some of the addresses the issue seemed to have gone away. What is weird that those addresses are not listed in any of the guidelines provided by Microsoft and we still didn't get a good answer from their support as to the reason they are actually needed. We are currently using Azure App Service Environment v2 and will probably migrate to v3. However not sure if this would help.

user_27254870
Related