Implementing C#/.NET gRPC Healthcheck in kubernetes

Viewed 1141

Goal

I want to add a healthcheck to a .NET microservice using the gRPC health probe protocol.

The microservice is nothing more than the Greeter service created with the dotnet new grpc template. I've gotten this to run and build locally, directly with Docker, and finally on my development K8s cluster.

Problem

Following this post and others, I have added the NuGet package to the project, and mapped the endpoint just like with the greeter service; however, the liveness/readiness probes fail due to timeout. I'm new to C#/.NET, so I expect that i've made a mistake in that area, but it is possible that the problem is in the kubernetes resources, too.

Code

Startup.cs

using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Http;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Grpc.HealthCheck;

namespace Company.Project.GRPC
{
    public class Startup
    {
        // This method gets called by the runtime. Use this method to add services to the container.
        // For more information on how to configure your application, visit https://go.microsoft.com/fwlink/?LinkID=398940
        public void ConfigureServices(IServiceCollection services)
        {
            services.AddGrpc();
            services.AddSingleton<HealthServiceImpl>(); // <-- makes sense to me generally though the details are hazy 
        }

        // This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
        public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
        {
            if (env.IsDevelopment())
            {
                app.UseDeveloperExceptionPage();
            }

            app.UseRouting();

            app.UseEndpoints(endpoints =>
            {
                // Healthcheck services
                endpoints.MapGrpcService<HealthServiceImpl>(); // <-- what besides this is needed?
                
                // App gRPC services
                endpoints.MapGrpcService<GreeterService>();

                endpoints.MapGet("/", async context =>
                {
                    await context.Response.WriteAsync("Communication with gRPC endpoints must be made through a gRPC client. To learn how to create a client, visit: https://go.microsoft.com/fwlink/?linkid=2086909");
                });
            });

        }
    }
}

Program.cs

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Hosting;
using Microsoft.Extensions.Hosting;
using Microsoft.AspNetCore.Server.Kestrel.Core;

namespace Company.Project.GRPC
{
    public class Program
    {
        public static void Main(string[] args)
        {
            CreateHostBuilder(args).Build().Run();
            
        }

        // Additional configuration is required to successfully run gRPC on macOS.
        // For instructions on how to configure Kestrel and gRPC clients on macOS, visit https://go.microsoft.com/fwlink/?linkid=2099682
       public static IHostBuilder CreateHostBuilder(string[] args) =>
        Host.CreateDefaultBuilder(args)
            .ConfigureWebHostDefaults(webBuilder =>
            {
                webBuilder.ConfigureKestrel(options =>
                {
                    // Setup a HTTP/2 endpoint without TLS.
                    // TODO: Review security. May in fact be ok as an internal-only service
                    // options.ListenLocalhost(5000, o => o.Protocols = 
                    //     HttpProtocols.Http2);
                    options.Listen( System.Net.IPAddress.Parse("0.0.0.0"), 5000);
                    options.Listen( System.Net.IPAddress.Parse("0.0.0.0"), 50051);
                });
                webBuilder.UseStartup<Startup>();

            });
    }
}

Deployment.yaml

...
          ports:
            - containerPort: 5000
            - containerPort: 50051
          readinessProbe:
            exec:
              command:
                - /bin/grpc_health_probe
                - -addr=:50051
            initialDelaySeconds: 15
            periodSeconds: 5
            timeoutSeconds: 5
            successThreshold: 1
            failureThreshold: 1
          livenessProbe:
            exec:
              command:
                - /bin/grpc_health_probe
                - -addr=:50051   # <-- doesn't matter which port I use
            initialDelaySeconds: 15
            periodSeconds: 5
            timeoutSeconds: 5
            successThreshold: 1
            failureThreshold: 1

Behavior

When I tail the logs for this pod, I see the server start up just fine, but then get killed after the liveness probe fails. The pod Events shows: Container myservice failed liveness probe, will be restarted

The probe client is the official one distributed from the link at the top. I've checked that the file is there, and that it can be executed as instructed.

I don't need any functionality beyond the basic Check that the server is reachable and running, so I believe that the step of subclassing HealthCheckImpl that is included in the post I referenced is superfluous.

What do I need to do/understand to get a working healthcheck on this simple service?

0 Answers
Related