Istio envoy proxy request loop causing OOM

Viewed 806

I have an interesting problem. I think I've found an infinite request loop that's causing my istio-proxy to crash with an OOM error in a specific circumstance.

When I submit the request locally to the app directly from inside the application container it seems to work fine and in the istio-proxy logs I see that the upstream_cluster is PassthroughCluster

However when I submit the request through the envoy proxy I'm using for SSL termination (not the istio-proxy for the app/mesh) the request seems to be retried/repeated until the istio-proxy crashes with an OOM; the only upstream_cluster I see in those logs is inbound|80|myapp

Has anyone see something like this before? It seems like the request is being routed to the wrong listener or the envoy-proxy we have sitting in front of the app that's in the mesh is somehow messing up the request.


A few more details about the symptom:

I have just one app in the service mesh and the traffic coming to that app comes from an envoy proxy that's handling SSL termination

Envoy -> (istio-proxy / app ) -> other services

When I send the request directly to localhost from within app it works fine, when I port forward directly to the (istio-proxy / app) pod and issue the request it also appears to work fine. When I send the request through Envoy I start to see thousands of identical requests in the (istio-proxy / app) logs and eventually the istio-proxy crashes with OOM.

1 Answers

It turns out that this wasn't actually an issue with Istio or Envoy;

My application was acting as a reverse proxy and forwarding inbound requests.

When the request came in through Envoy it changed the HOST header of the request to match the k8s service.

When our service received the request it forwarded it with the host header intact and then the istio-sidecar tried to use this header; this is what caused the issue.

Updating or removing the host header in the reverse proxy app fixes the issue.

This github issue has more details

https://github.com/istio/istio/issues/26946#issuecomment-715365117

Related