504 error on nodejs sails API server randomly across all endpoints

Viewed 128

we have sails js as an API server and intermittently we are facing 504 upstream timed out the issue, the call was going till Nginx from there its throwing upstream timeout, there is no log coming on the application server, and it is happening across all the APIs intermittently not all the time, and the traffic on server is very low i.e. 5-10 requests per minute and there is no heavy computation going on, so not sure how can I debug this issue. this is a very random issue. also, there are No server restarts or any other errors on application logs. it's running fine on PM2. we are using an AWS EC2 instance. current timeout is 60 seconds but none of our APIs takes more than 500 milliseconds. we are using the node 6.6 version as it is a legacy monolith app so can not upgrade it due to multiple dependencies and no single owner. and requests are passing through the load balancer to NGINX but some time does not reach the application server. also, the instance size is quite bigger in terms of CPU and memory and traffic is extremely low. this is a very random behavior not specific to API. sometimes it can happen 1 out of 10 sometimes 2 out of 5 requests can through a gateway timeout issue.

some of the logs from Nginx are below-

[error] 31688#31688: *38998779 connect() failed (110: Connection timed out) while connecting to upstream, client 10.X.X.X

2022/04/22 16:36:37 [error] 31690#31690: *38998991 connect() failed (110: Connection timed out) while connecting to upstream, client: <server_ip>, server: <DNS_URL>

guys I have tried almost all the things from StackOverflow but nothing helping me, so please help me to find the root cause so I can mitigate the issue

0 Answers
Related