.NET6 gRPC fails intermittently

user_3621968 0
Viewed 472

My setup:

  1. A Kubernetes cluster with a gRPC server. Currently 1 replica. An nginx ingress in front of it. Running on ASPNET-6.0-bullseye-slim image.
  2. A C# client with .NET6 on Windows and on Linux

My client just makes calls to the server, the same endpoint, 100s of times in parallel. Most (95%) of the calls succeed but then some of them fail with the following errors (I see the errors regardless of connecting via the ingress or directly via IP):

ERROR 1: Client-side:

Status(StatusCode="Unavailable", Detail="Error starting gRPC call. HttpRequestException: An error occurred while sending the request. IOException: The response ended prematurely while waiting for the next frame from the server.", DebugException="System.Net.Http.HttpRequestException: An error occurred while sending the request.
---> System.IO.IOException: The response ended prematurely while waiting for the next frame from the server.
at System.Net.Http.Http2Connection.<ReadFrameAsync>g__ThrowMissingFrame|57_1()
at System.Net.Http.Http2Connection.ReadFrameAsync(Boolean initialFrame)
at System.Net.Http.Http2Connection.ProcessIncomingFramesAsync()
at System.Net.Http.Http2Connection.SendHeadersAsync(HttpRequestMessage request, CancellationToken cancellationToken, Boolean mustFlush)
at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at Grpc.Net.Client.Balancer.Internal.BalancerHttpHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Grpc.Net.Client.Internal.GrpcCall`2.RunCall(HttpRequestMessage request, Nullable`1 timeout)")

Server Side:

Exception":"System.IO.IOException: The request stream was aborted. 
---\u003E Microsoft.AspNetCore.Connections.ConnectionAbortedException: The HTTP/2 connection faulted. 
--- End of inner exception stack trace --- at
System.IO.Pipelines.Pipe.GetReadResult(ReadResult\u0026 result) at
System.IO.Pipelines.Pipe.GetReadAsyncResult() at
Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2MessageBody.ReadAsync(CancellationToken cancellationToken) at
System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder\u00601.StateMachineBox\u00601.System.Threading.Tasks.Sources.IValueTaskSource\u003CTResult\u003E.GetResult(Int16 token) at
Grpc.AspNetCore.Server.Internal.PipeExtensions.ReadSingleMessageAsync[T](PipeReader input, HttpContextServerCallContext serverCallContext, Func\u00602 deserializer) at
Grpc.AspNetCore.Server.Internal.CallHandlers.UnaryServerCallHandler\u00603.HandleCallAsyncCore(HttpContext httpContext, HttpContextServerCallContext serverCallContext) at
Grpc.AspNetCore.Server.Internal.CallHandlers.ServerCallHandlerBase\u00603.\u003CHandleCallAsync\u003Eg__AwaitHandleCall|8_0(HttpContextServerCallContext serverCallContext, Method\u00602 method, Taskver:

ERROR 2: Client-Side:

Msg: Status(StatusCode="Internal", Detail="Error starting gRPC call. HttpRequestException: An error occurred while sending the request. IOException: The request was aborted. Http2StreamException: The HTTP/2 server reset the stream. HTTP/2 error code 'PROTOCOL_ERROR' (0x1).", DebugException="System.Net.Http.HttpRequestException: An error occurred while sending the request.
---> System.IO.IOException: The request was aborted.
---> System.Net.Http.Http2StreamException: The HTTP/2 server reset the stream. HTTP/2 error code 'PROTOCOL_ERROR' (0x1).
--- End of inner exception stack trace ---
at System.Net.Http.Http2Connection.ThrowRequestAborted(Exception innerException)
at System.Net.Http.Http2Connection.Http2Stream.CheckResponseBodyState()
at System.Net.Http.Http2Connection.Http2Stream.TryEnsureHeaders()
at System.Net.Http.Http2Connection.Http2Stream.ReadResponseHeadersAsync(CancellationToken cancellationToken)
at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at Grpc.Net.Client.Balancer.Internal.BalancerHttpHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Grpc.Net.Client.Internal.GrpcCall`2.RunCall(HttpRequestMessage request, Nullable`1 timeout)")

Server Side:

Exception":"Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2ConnectionErrorException: HTTP/2 connection error (PROTOCOL_ERROR): Invalid HTTP/2 connection preface. at
Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2Connection.ParsePreface(ReadOnlySequence\u00601\u0026 buffer, SequencePosition\u0026 consumed, SequencePosition\u0026 examined) at
Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2Connection.TryReadPrefaceAsync() at
Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2Connection.ProcessRequestsAsync[TContext](IHttpApplication\u00601 application)","State":{"Message":"Connection id \u00220HMH7NHUNIR2E\u0022: HTTP/2 connection error.","ConnectionId":"0HMH7NHUNIR2E","{OriginalFormat}":"Connection id \u0022{ConnectionId}\u0022: HTTP/2 connection error.

My server app settings.json configuration is this:

{
  "Kestrel": {
    "Endpoints": {
      "Grpc": {
        "Url": http://+:8000,
        "Protocols": "Http2"
      },
      "Http": {
        "Url": http://+:7999,
        "Protocols": "Http1"
      }
    },
  },

Startup.cs is this:

public void ConfigureServices(IServiceCollection services)
{
    services.AddResponseCompression(options => options.EnableForHttps = true);
    AppContext.SetSwitch("System.Net.Http.SocketsHttpHandler.Http2UnencryptedSupport", true);
    AppContext.SetSwitch("System.Net.Http.SocketsHttpHandler.Http2Support", true);
    services.AddGrpc(options =>
    {
        options.Interceptors.Add<CustomRequestInterceptor>();
    });

    services.AddApiVersioning();
    services.AddAuthorization();
    services.AddGrpcHttpApi();
   
    services.AddCors(options =>
    {
        options.AddPolicy(
            CorsPolicy,
            builder => builder.AllowAnyOrigin().AllowAnyHeader().AllowAnyMethod());
        });
    }
}
    
public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
    app.UseResponseCompression();
    app.UseRouting();
    app.UseAuthentication();
    app.UseAuthorization();
    app.UseCors(CorsPolicy);

    app.UseEndpoints(endpoints =>
    {
        endpoints.MapGrpcService<GrpcService1Imp>();
        endpoints.MapGrpcService<GrpcService2Imp>();   
    });
}

Client side code:

var channel = GrpcChannel.ForAddress(url);
var client = new GrpcService1Client(channel);
client.callfunction();
.net-6.0 grpc asp.net
0 Answers
Related