Bad Gateway 502 error with Apache mod_proxy and Tomcat

Just to add some specific settings, I had a similar setup (with Apache 2.0.63 reverse proxying onto Tomcat 5.0.27).

For certain URLs the Tomcat server could take perhaps 20 minutes to return a page.

I ended up modifying the following settings in the Apache configuration file to prevent it from timing out with its proxy operation (with a large over-spill factor in case Tomcat took longer to return a page):

Timeout 5400
ProxyTimeout 5400

Some backgound

ProxyTimeout alone wasn't enough. Looking at the documentation for Timeout I'm guessing (I'm not sure) that this is because while Apache is waiting for a response from Tomcat, there is no traffic flowing between Apache and the Browser (or whatever http client) - and so Apache closes down the connection to the browser.

I found that if I left the Timeout setting at its default (300 seconds), then if the proxied request to Tomcat took longer than 300 seconds to get a response the browser would display a "502 Proxy Error" page. I believe this message is generated by Apache, in the knowledge that it's acting as a reverse proxy, before it closes down the connection to the browser (this is my current understanding - it may be flawed).

The proxy error page says:

Proxy Error

The proxy server received an invalid response from an upstream server. The proxy server could not handle the request GET.

Reason: Error reading from remote server

...which suggests that it's the ProxyTimeout setting that's too short, while investigation shows that Apache's Timeout setting (timeout between Apache and the client) that also influences this.

So, answering my own question here. We ultimately determined that we were seeing 502 and 503 errors in the load balancer due to Tomcat threads timing out. In the short term we increased the timeout. In the longer term, we fixed the app problems that were causing the timeouts in the first place. Why Tomcat timeouts were being perceived as 502 and 503 errors at the load balancer is still a bit of a mystery.

I'm guessing your using mod_proxy_http (or proxy balancer).

Look in your tomcat logs (localhost.log, or catalina.log) I suspect your seeing an exception in your web stack bubbling up and closing the socket that the tomcat worker is connected to.

I know this does not answer this question, but I came here because I had the same error with nodeJS server. I am stuck a long time until I found the solution. My solution just adds slash or /in end of proxyreserve apache.

my old code is:

ProxyPass / http://192.168.1.1:3001
ProxyPassReverse / http://192.168.1.1:3001

the correct code is:

ProxyPass / http://192.168.1.1:3001/
ProxyPassReverse / http://192.168.1.1:3001/

If you want to handle your webapp's timeout with an apache load balancer, you first have to understand the different meaning of timeout. I try to condense the discussion I found here: http://apache-http-server.18135.x6.nabble.com/mod-proxy-When-does-a-backend-be-considered-as-failed-td5031316.html :

It appears that mod_proxy considers a backend as failed only when the transport layer connection to that backend fails. Unless failonstatus/failontimeout is used. ...

So, setting failontimeout is necessary for apache to consider a timeout of the webapp (e.g. served by tomcat) as a fail (and consecutively switch to the hot spare server). For the proper configuration, note the following misconfiguration:

ProxyPass / balancer://localbalance/ failontimeout=on timeout=10 failonstatus=50

This is a misconfiguration because:

You are defining a balancer here, so the timeout parameter relates to the balancer (like the two others). However for a balancer, the timeout parameter is not a connection timeout (like the one used with BalancerMember), but the maximum time to wait for a free worker/member (e.g. when all the workers are busy or in error state, the default being to not wait).

So, a proper configuration is done like this

1. set timeout at the BalanceMember level:

 <Proxy balancer://mycluster>
     BalancerMember http://member1:8080/svc timeout=6 
 ... more BalanceMembers here
</Proxy>

2. set the failontimeout on the balancer

ProxyPass /svc balancer://mycluster failontimeout=on

Restart apache.