GRPc-netty upstream connect error or disconnect/reset before headers

user_8231671 0
Viewed 2730

I'm trying to use GRPC between my services. I have just three services. Two of them running in k8s. One in GCP. I make calls from the service in GCP to other services. I use the same NettyChannelBuilder for my stubs. The clients have the same requests per seconds rate. I've set keepAliveWithoutCalls=true, keepAliveTime, idleTimeout in the builder. Unfortunately I periodically get this error from only one client? 

io.grpc.StatusRuntimeException: UNAVAILABLE: upstream connect error or disconnect/reset before headers. reset reason: connection failure
    at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233)
    at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214)

Could anyone please help me to find solution? I've read many issues on github already. Now, I'm going to set retry policy for calls but it's not the best solution.

grpc kubernetes netty java
1 Answers

From github

We faced the same error. The reason for us was because tcp_keepalive was set too high for our upstream service. We changed keepalive_time to 300 seconds and the problem went away.

It makes sense in our case because we have Envoy pointing to a network load balancer (aws) which has a 350s idle timeout.

We added this to our envoy config.

clusters:
    - name: grpc-service
      connect_timeout: 0.25s
      http2_protocol_options: {}
      upstream_connection_options:
        tcp_keepalive:
          keepalive_time: 300
user_114718090
Related