Nginx PHP-FPM Stops Working, open socket left in connection

user_456850 0
Viewed 6560

I have this re-occurring issue with Nginx about once a day, about twice under high traffic loads. The fix is easy, restart the server, but when the error happens, Nginx completely stops working. I have an NGINX PHP-FPM setup.

It issue starts with:

020/09/27 09:57:27 [error] 38#38: *430982 upstream timed out (110: Connection timed out) while connecting to upstream, client: x.x.x.x, server: example.com, request: "POST /api/sessions/wri

And then it progress into:

2020/09/27 10:03:22 [alert] 40#40: *431277 open socket #18 left in connection 51
2020/09/27 10:03:22 [alert] 38#38: *431298 open socket #34 left in connection 166
2020/09/27 10:03:22 [alert] 40#40: *431288 open socket #28 left in connection 59
2020/09/27 10:03:22 [alert] 38#38: *431296 open socket #32 left in connection 169
2020/09/27 10:03:22 [alert] 38#38: *431257 open socket #36 left in connection 177
2020/09/27 10:03:22 [alert] 38#38: *431291 open socket #23 left in connection 178
2020/09/27 10:03:22 [alert] 38#38: *431253 open socket #27 left in connection 188
2020/09/27 10:03:22 [alert] 38#38: *431300 open socket #31 left in connection 197
2020/09/27 10:03:22 [alert] 38#38: *431312 open socket #12 left in connection 204
2020/09/27 10:03:22 [alert] 38#38: *431259 open socket #38 left in connection 206
2020/09/27 10:03:22 [alert] 37#37: aborting
2020/09/27 10:03:22 [alert] 38#38: aborting
2020/09/27 10:03:22 [alert] 40#40: aborting
2020/09/27 10:03:23 [warn] 21568#21568: 8096 worker_connections exceed open file resource limit: 1024
2020/09/27 10:08:24 [warn] 21574#21574: *636 upstream server temporarily disabled while connecting to upstream,

Now GET requests still do work. So if I go to the website, it loads. But anything that is POST, PUT or DELETE will fail, so ultimately the users cant do anything but browse.

Any ideas on why this is happening? And is there a healthcheck that can be used to detect these issues?

nginx php
1 Answers

Probably that your Nginx process is opening too many files. You can increase this limit by following this article: How to Increase Number of Open Files Limit in Linux

To change this setting, you need the root privilege and do as follows:

  1. Open /etc/security/limits.conf file: vi /etc/security/limits.conf

  2. Append those lines at the end of the file:

## Example hard limit for max opened files
*    hard    nofile    10240
## Example soft limit for max opened files
*    soft    nofile    10240
  1. Save all changes and Reboot your Server/VPS to apply new settings.

Note: The number 10240 is the maximum number of files that a process can open. Change this value as your needs.

user_48585720
Related