AWS HTTP API Gateway 503 Service Unavailable

user_4368450 0
Viewed 1558

I have an HTTP API Gateway with a HTTP Integration backend server on EC2. The API has lots of queries during the day and looking at the logs i realized that the API is returning sometimes a 503 HTTP Code with a body:

{ "message": "Service Unavailable" }

When i found out this, i tried the API and running the HTTP requests many times on Postman, when i try twenty times i get at least one 503.

I then thought that the HTTP Integration Server was busy but the server is not loaded and i tried going directly to the HTTP Integration Server and i get 200 responses all the times.

The timeout parameter is set to 30000ms and the endpoint average response time is 200ms so timeout is not a problem. Also the HTTP 503 is not after 30 seconds of the request but instantly.

Can anyone help me?

Thanks

aws-api-gateway amazon-web-services amazon-ec2
2 Answers

I solved this issue by editing the keep-alive connection parameters of my internal integration server. The AWS API Gateway needs the keep alive parameters on a standard configuration, so I started tweaking my NGINX server parameters until I solved the issue.

user_43684500

Had the same issue on a selfmade Microservice with Node that was integrated into AWS API-Gateway. After some reconfiguration of the Cloudwatch-Logs I got further indicator on what is wrong: INTEGRATION_NETWORK_FAILURE

Verify your problem is alike - i.e. through elaborated log output

In API-Gateway - Logging add more output in "Log format" enter image description here Use this or similar content for "Log format":

{"httpMethod":"$context.httpMethod","integrationErrorMessage":"$context.integrationErrorMessage","protocol":"$context.protocol","requestId":"$context.requestId","requestTime":"$context.requestTime","resourcePath":"$context.resourcePath","responseLength":"$context.responseLength","routeKey":"$context.routeKey","sourceIp":"$context.identity.sourceIp","status":"$context.status","errMsg":"$context.error.message","errType":"$context.error.responseType","intError":"$context.integration.error","intIntStatus":"$context.integration.integrationStatus","intLat":"$context.integration.latency","intReqID":"$context.integration.requestId","intStatus":"$context.integration.status"}

After using API-Gateway Endpoint and failing consult the logs again - should be looking like that: enter image description here

Solve in NodeJS Microservice (using Express)

Add timeouts for headers and keep-alive on express servers socket configuration when upon listening.

const app = require('express')();

// if not already set and required to advertise the keep-alive through HTTP-Response you might want to use this
/* 
app.use((req: Request, res: Response, next: NextFunction) => {
    res.setHeader('Connection', 'keep-alive');
    res.setHeader('Keep-Alive', 'timeout=30');
    next();
});
*/

/* ..you r main logic.. */

const server = app.listen(8080, 'localhost', () => {
    console.warn(`⚡️[server]: Server is running at http://localhost:8080`);
});

server.keepAliveTimeout = 30 * 1000; // <- important lines
server.headersTimeout = 35 * 1000; // <- important lines

Reason

Some AWS Components seem to demand a connection kept alive - even if server responding otherwise (connection: close). Upon reusage in API Gateway (and possibly AWS ELBs) the recycling will fail because other-side most likely already closed hence the assumed "NETWORK-FAILURE".

This error seems intermittent - since at least the API-Gateway seems to close unused connections after a while providing a clean execution the next time. I can only assume they do that for high-performance and not divert to anything less.

user_31147020
Related