gRPC client in C# does not work with gRPC server in Go with mTLS support

I have a gRPC server in Golang that has mTLS enabled using the following ServerOptions:

// getServerOptions returns a list of GRPC server options.
// Current options are TLS certs and opencensus stats handler.
func (h *serviceHandler) getServerOptions() []grpc.ServerOption {
    tlsCer, err := tls.LoadX509KeyPair(tlsDir+"tls.crt", tlsDir+"tls.key")
    if err != nil {
        logger.WithError(err).Fatal("failed to generate credentials")
    }

    cfg := &tls.Config{
        Certificates: []tls.Certificate{tlsCer},
        ClientAuth:   tls.RequireAndVerifyClientCert,
        GetConfigForClient: func(*tls.ClientHelloInfo) (*tls.Config, error) {
            h.certMutex.RLock()
            defer h.certMutex.RUnlock()
            return &tls.Config{
                Certificates: []tls.Certificate{tlsCer},
                ClientAuth:   tls.RequireAndVerifyClientCert,
                ClientCAs:    h.caCertPool,
            }, nil
        },
    }
    // Add options for creds and OpenCensus stats handler to enable stats and tracing.
    return []grpc.ServerOption{grpc.Creds(credentials.NewTLS(cfg)), grpc.StatsHandler(&ocgrpc.ServerHandler{})}
}

The server works fine for a gRPC client in Golang, but fails for the following gRPC c# client after the cert exchange handshake.

        static async Task Main(string[] args)
        {
            string baseAddress = "x.x.x.x";
            var x509Cert = new X509Certificate2("client.pfx", "123");
            var client = CreateClientWithCert("https://" + baseAddress + ":443", x509Cert);

            try {
                var response = await client.PostAllocateAsync(new AllocationRequest {Namespace = "Default"});
                Console.Write(response.State.ToString());
            } 
            catch(RpcException e)
            {
                Console.WriteLine($"gRPC error: {e.Status.Detail}");
            }
            catch 
            {
                Console.WriteLine($"Unexpected error calling agones-allocator");
                throw;
            }
        }

        public static AllocationService.AllocationServiceClient CreateClientWithCert(
            string baseAddress,
            X509Certificate2 certificate)
        {

            var loggerFactory = LoggerFactory.Create(logging =>
            {
                logging.AddConsole();
                logging.SetMinimumLevel(LogLevel.Trace);
            });

            // Add client cert to the handler
            var handler = new HttpClientHandler();
            handler.ClientCertificates.Add(certificate);
            handler.ServerCertificateCustomValidationCallback = 
                HttpClientHandler.DangerousAcceptAnyServerCertificateValidator;

            // Create the gRPC channel
            var channel = GrpcChannel.ForAddress(baseAddress, new GrpcChannelOptions
            {
                HttpClient = new HttpClient(handler),
                LoggerFactory = loggerFactory,
            });

            return new AllocationService.AllocationServiceClient(channel);
        }
    }

Here is the trace log:

dbug: Grpc.Net.Client.Internal.GrpcCall[1]
      Starting gRPC call. Method type: 'Unary', URI: 'https://x.x.x.x/v1alpha1.AllocationService/PostAllocate'.
dbug: Grpc.Net.Client.Internal.GrpcCall[18]
      Sending message.
trce: Grpc.Net.Client.Internal.GrpcCall[21]
      Serialized 'V1Alpha1.AllocationRequest' to 9 byte message.
trce: Grpc.Net.Client.Internal.GrpcCall[19]
      Message sent.
fail: Grpc.Net.Client.Internal.GrpcCall[6]
      Error starting gRPC call.
System.Net.Http.HttpRequestException: An error occurred while sending the request.
 ---> System.IO.IOException: The response ended prematurely.
   at System.Net.Http.HttpConnection.FillAsync()
   at System.Net.Http.HttpConnection.ReadNextResponseHeaderLineAsync(Boolean foldedHeadersAllowed)
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
   at Grpc.Net.Client.Internal.GrpcCall`2.RunCall(HttpRequestMessage request)
dbug: Grpc.Net.Client.Internal.GrpcCall[8]
      gRPC call canceled.
fail: Grpc.Net.Client.Internal.GrpcCall[3]
      Call failed with gRPC error status. Status code: 'Internal', Message: 'Error starting gRPC call: An error occurred while sending the request.'.
dbug: Grpc.Net.Client.Internal.GrpcCall[4]
      Finished gRPC call.
gRPC error: Error starting gRPC call: An error occurred while sending the request.

Can someone please help me understand what is the reason for the failure? Using SslCredentials also fails. x.x.x.x is the IP replacement for the privacy reasons.