We are using a webserver with ElasticBeanstalk from 2019.,
the platform is
tomcat 8.5 with java8 running on 64 bit Amazon Linux. httpd as proxy
recently (from Jan 30th) we started getting Service Unavailable issues if go to the endpoint from time to time. and if we refresh 2-3 times it will get resolved on its own.
then I download full logs. under elasticbeanstalk-error_log I can see
[Mon Feb 28 10:00:58.338035 2022] [proxy:error] [pid 14882:tid 139757313533696] (13)Permission denied: AH02454: HTTP: attempt to connect to Unix domain socket /var/run/httpd/ (localhost) failed
[Mon Feb 28 10:00:58.338078 2022] [proxy_http:error] [pid 14882:tid 139757313533696] [client <private-ip-here>:12566] AH01114: HTTP: failed to make connection to backend: httpd-UDS, referer: http://<custom-end-point>/1/<name.jsp>?s=sec$$4P!&refresh=300
[Mon Feb 28 10:43:40.663468 2022] [proxy:error] [pid 14882:tid 139757120071424] (13)Permission denied: AH02454: HTTP: attempt to connect to Unix domain socket /var/run/httpd/ (localhost) failed
[Mon Feb 28 10:43:40.663518 2022] [proxy_http:error] [pid 14882:tid 139757120071424] [client <private-ip-here>:21136] AH01114: HTTP: failed to make connection to backend: httpd-UDS
repeated multiple times from Jan30th.
and when I look at access.log
I can see 503 error log exactly at the same time when permission denied error logs in elasticbeanstalk-error_log
And I looked at the running process using ps -aux | grep HTTPd and ps -aux | grep tomcat
both are running from 2019 and have no restarts.
what more I can do to troubleshoot these issuesWe are running a web application written in Java(tomcat8) hosted in AWS ElastcBeanStalk
Some weeks back we started getting 503 error randomly
When we checked the elasticbeanstalk-erorr_logs [Thu Mar 03 13:22:12.906144 2022] [proxy:error] [pid 14882:tid 139757338711808] (13)Permission denied: AH02454: HTTP: attempt to connect to Unix domain socket /var/run/httpd/ (localhost) failed [Thu Mar 03 13:22:12.906202 2022] [proxy_http:error] [pid 14882:tid 139757338711808] [client 172.31.17.0:61382] AH01114: HTTP: failed to make connection to backend: httpd-UDS, referer: http://our-domain.com/1/callBackLog.jsp
The error logs are suggesting connection error with backend unix socket When we checked in /var/run/httpd/ folder, there were no unix sockets(.sock files)
But in apache httpd config <VirtualHost *:80> <Proxy *> Require all granted ProxyPass / http://localhost:8080/ retry=0 ProxyPassReverse / http://localhost:8080/ ProxyPreserveHost on
ErrorLog /var/log/httpd/elasticbeanstalk-error_log
the proxy backend is ip address not unix socket
As per the config httpd should connect to backend ip address(localhost:8080) but why is it complaining about unix socket
Have anyone faced similar issues?
============= UPDATE
The error logs are suggesting connection error with backend unix socket
When we checked in /var/run/httpd/ folder, there were no unix sockets(.sock files)
But in apache httpd config
<VirtualHost *:80> <Proxy *> Require all granted ProxyPass / http://localhost:8080/ retry=0 ProxyPassReverse / http://localhost:8080/ ProxyPreserveHost on
ErrorLog /var/log/httpd/elasticbeanstalk-error_log the proxy backend is ip address not unix socket
As per the config httpd should connect to backend ip address(localhost:8080) but why is it complaining about unix socket
Have anyone faced similar issues?