I'm running a .NET Core gRPC server where I'm experiencing a memory leaks. Currently I'm using version 1.22.1 of Grpc based on the c based library.
The following error is often triggered:
W1027 Grpc.Core.Server Exception while handling RPC. `System.InvalidOperationException: Operation is not valid due to the current state of the object. at Grpc.Core.Internal.AsyncCallServer. SendStatusFromServerAsync(Status status, Metadata trailers, Nullable1 optionalWrite) at Grpc.Core.Internal.DuplexStreamingServerCallHandler2.HandleCall(ServerRpcNew newRpc, CompletionQueueSafeHandle cq) at Grpc.Core.Server.HandleCallAsync(ServerRpcNew newRpc, CompletionQueueSafeHandle cq, Action continuation)
and
E1027 140088109414144 /var/local/git/grpc/src/core/ext/transport/chttp2/transport/chttp2_transport.cc:2823: ipv4:10.132.15.236:6845: Keepalive watchdog fired. Closing transport.
Below is a picture of how the memory leak evolves it starts pretty fast and then slows down when the application is near it's maximum memory limit, however, it does stop and gets shut down due to exceeding the memory usage.
Can those errors be related to the memory leak somehow ?
UPDATE:
The issue seems to be related to idle TCP connections. By default, idle TCP connections are not removed in the gRPC version we are using. by settings the following option "grpc.max_connection_idle_ms" to a few minutes, the number of TCP connections stops rising and thereby the memory leak has decreased significantly.