Elegant way to handle goroutine in IO wait state in GRPC

Question

Our server (grpc-gateway + grpc) is running on K8S with go 1.13, and terminate stack information

    Last State:    Terminated
      Reason:      Error
      Message:     o.(*Reader).fill(0xc002ec78c0)
                   /Users/local/go/src/bufio/bufio.go:100 +0x103
bufio.(*Reader).Peek(0xc002ec78c0, 0x4, 0x0, 0x0, 0x0, 0x0, 0xc002e79ad0)
  /Users/local/go/src/bufio/bufio.go:138 +0x4f
net/http.(*conn).readRequest(0xc00245a000, 0x1ef4800, 0xc000be7780, 0x0, 0x0, 0x0)
  /Users/local/go/src/net/http/server.go:962 +0xb3b
net/http.(*conn).serve(0xc00245a000, 0x1ef4800, 0xc000be7780)
  /Users/local/go/src/net/http/server.go:1817 +0x6d4
created by net/http.(*Server).Serve
  /Users/local/go/src/net/http/server.go:2928 +0x384

goroutine 8724981 [IO wait]:
internal/poll.runtime_pollWait(0x7f5d3a8f84e8, 0x72, 0xffffffffffffffff)
  /Users/local/go/src/runtime/netpoll.go:184 +0x55
internal/poll.(*pollDesc).wait(0xc000155d98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
  /Users/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45
internal/poll.(*pollDesc).waitRead(...)
  /Users/local/go/src/internal/poll/fd_poll_runtime.go:92
internal/poll.(*FD).Read(0xc000155d80, 0xc00138b000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
  /Users/local/go/src/internal/poll/fd_unix.go:169 +0x1cf
net.(*netFD).Read(0xc000155d80, 0xc00138b000, 0x1000, 0x1000, 0xc00142b9e8, 0x4ce13d, 0xc000155d80)
  /Users/local/go/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc0000dd690, 0xc00138b000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
  /Users/local/go/src/net/net.go:184 +0x68
net/http.(*connReader).Read(0xc001e9b050, 0xc00138b000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
  /Users/local/go/src/net/http/server.go:785 +0xf4
bufio.(*Reader).fill(0xc0021fe060)
  /Users/local/go/src/bufio/bufio.go:100 +0x103
bufio.(*Reader).Peek(0xc0021fe060, 0x4, 0x0, 0x0, 0x0, 0x0, 0xc00142bad0)
  /Users/local/go/src/bufio/bufio.go:138 +0x4f
net/http.(*conn).readRequest(0xc0008f43c0, 0x1ef4800, 0xc000503640, 0x0, 0x0, 0x0)
  /Users/local/go/src/net/http/server.go:962 +0xb3b
net/http.(*conn).serve(0xc0008f43c0, 0x1ef4800, 0xc000503640)
  /Users/local/go/src/net/http/server.go:1817 +0x6d4
created by net/http.(*Server).Serve
  /Users/local/go/src/net/http/server.go:2928 +0x384

      Exit Code:    2

Per this question, one possible solution is

>  s := new(http.Server)
>  // ...
>  s.ReadTimeout = 5 * time.Second
>  s.WriteTimeout = 5 * time.Second
>  // ...

However, we failed to find the ReadTimeout from grpc.NewServer, is there anything are we missing? or how to handle goroutine in IO wait state in GRPC more elegantly?

grpc version is v1.21.1

user_1218512 · Answer

gprc.NewServer allows for zero or more ServerOption during creation:

func NewServer(opt ...ServerOption) *Server

while there does not appear to be a ReadTimeout or WriteTimeout equivalent to http.Server you could try keepalive.ServerParameters:

type ServerParameters struct {
    MaxConnectionIdle     time.Duration // The current default value is infinity.
    MaxConnectionAge      time.Duration // The current default value is infinity.
    MaxConnectionAgeGrace time.Duration // The current default value is infinity.
    Time                  time.Duration // The current default value is 2 hours.
    Timeout               time.Duration // The current default value is 20 seconds.
}

(full docs keepalive.ServerParameters)

and tune say keepalive.ServerParameters.Time to something lower than 2 hours:

srv := grpc.NewServer(
    keepalive.ServerParameters{Time:5*time.Minute},
)

This will lower connection reuse - but will also free up client connections that are long since dead.

user_6444915 · Answer

You use Context with Timeout.

https://golang.org/pkg/context/

Elegant way to handle goroutine in IO wait state in GRPC

2 Answers