Summary:
I've been working on a Google Cloud Platform project for a while now and have recently started getting rather frequent, but seemingly random 502s back from the Google Cloud Load Balancer. I get these messages back when trying to access an App Engine flexible instance running nodejs. Split health checks (alive and ready) are configured for the app engine instance and are successful. The 502s never appear in the apps logs just in the load balancer logs. The app is set to auto scale with a minimum of two instances, and in this small test environment never has had enough load to actually need to add another instance. The two instances do not appear to be under any real load at all and so should respond without issue, but just by hitting a test page (which is very simple and does almost no work at all) I can generate 502s. The load balancer says that it cannot connect to the backend, but with the backend healthy and not busy I don't see how this could be the case. Running the code on a local system (which is sort of an apples to oranges comparison I realize) I do not see these errors or errors that might cause these.
Question:
How does one go about troubleshooting issues with the Google Load Balancer and App Engine?
Are there any suggested fixes or settings I could try?
Supporting screen shots:
Screen shots
1. load balancer log snippet
2. load balancer error detail
3. health check log snippet
4. instance CPU usage
5. instance memory usage
6. instance latency
7. error in browser
8. success in browser