.NET Core 3.1 - Worker Service in linux container: Clean Shutdown

Question

Below is the code from the base template in Visual Studio 2019 to create a Worker Service. When deploying this service to a linux container, the code CancellationToken in the ExecuteAsync method is never cancelled when the container is requested to stop. After stopping the container in various ways, I see 10 more "Worker running at" log entries and then the container stops. This is consistent with what I've read: docker sends a SIGTERM command, waits 10 seconds, then sends a SIGKILL command if the container doesn't stop on its own.

What I am searching for is an answer to how that CancellationToken is supposed to get set to Cancelled when the container is requested to stop? My end goal is to make sure any transactions in flight are completed or rolled back and other such cleanup work before the container is suddenly stopped.

public class Program
{
    public static void Main(string[] args)
    {
        CreateHostBuilder(args).Build().Run();
    }

    public static IHostBuilder CreateHostBuilder(string[] args) =>
        Host.CreateDefaultBuilder(args)
            .ConfigureServices((hostContext, services) =>
            {
                services.AddHostedService();
            });
}


public class Worker : BackgroundService
{
    private readonly ILogger _logger;

    public Worker(ILogger logger)
    {
        _logger = logger;
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            _logger.LogInformation("Worker running at: {time}", DateTimeOffset.Now);
            await Task.Delay(1000, stoppingToken);
        }
    }
}

I have tried subscribing to the AppDomain.CurrentDomain.ProcessExit event as that seems to be what most people around the web say to do (without any examples), but either the way I implemented it doesn't work or that just isn't the correct thing to do. Running the container from VS2019, the code of the event handler is never executed. Below is how I tried to implement that:

public static void Main(string[] args)
{
    using (CancellationTokenSource cts = new CancellationTokenSource())
    {
        AppDomain.CurrentDomain.ProcessExit += (sender, eventArgs) =>
        {
            cts.Cancel();
        };

        var host = CreateHostBuilder(args).Build();
        Task task;

        task = host.RunAsync(cts.Token);

        task.Wait(cts.Token);
    }
}

Update: I'm guessing the problem has something to do with the way template is building the docker file and starting the container. The SIGTERM is sent to PID 1 and looking at the results of ps -ef on the container, the calling of tail is PID 1 instead of anything the within the .netcore solution. I guess the next question would be how to fix this?

Update (Requested Docker File) The docker file is the default docker file as Visual Studio creates for this template.

#See https://aka.ms/containerfastmode to understand how Visual Studio uses this Dockerfile to build your images for faster debugging.

FROM mcr.microsoft.com/dotnet/runtime:5.0 AS base
WORKDIR /app

FROM mcr.microsoft.com/dotnet/sdk:5.0 AS build
WORKDIR /src
COPY ["WorkerService2/WorkerService2.csproj", "WorkerService2/"]
RUN dotnet restore "WorkerService2/WorkerService2.csproj"
COPY . .
WORKDIR "/src/WorkerService2"
RUN dotnet build "WorkerService2.csproj" -c Release -o /app/build

FROM build AS publish
RUN dotnet publish "WorkerService2.csproj" -c Release -o /app/publish

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "WorkerService2.dll"]

user_190638 · Answer

This is actually an issue with running the containers from Visual Studio inside of Docker for Windows (I'm guessing Docker for Mac will be similar) while debugging.

After reading through this link (https://aka.ms/containerfastmode) that is in the generated dockerfile last night, when running the images from Visual Studio to test, the entry point will not be the netcore application, so PID1 will not be the .netcore app and it will not receive the SIGTERM signal. However, the .netcore app should be the entry point when not started via Visual Studio.

The entry point is tail -f /dev/null, which is an infinite wait to keep the container running. When the app is launched through the debugger, it is the debugger that is responsible to run the app (that is, dotnet webapp.dll). If launched without debugging, the tooling runs a docker exec -i {containerId} dotnet webapp.dll to run the app.

After doing a build in Visual Studio, I tested the image with the :latest tag for my application from Docker for Windows directly. When I did this, the ENTRYPOINT in the dockerfile was used and PID 1 was the .netcore application as expected. Upon stopping the container, the CancellationToken.Cancel() was triggered and an OperationCancelledException was thrown.

Below is basic code for the template that I'd use going forward in order to account for the CancellationToken getting Cancelled and throwing an exception when it does.

{
    try
    {
        _logger.LogInformation($"WORKER STARTING at: {DateTimeOffset.Now}");

        while (!stoppingToken.IsCancellationRequested)
        {
            _logger.LogInformation("$Worker running at: {DateTimeOffset.Now}");
            await Task.Delay(1000, stoppingToken);
        }

        _logger.LogInformation($"Exited While Loop: {DateTimeOffset.Now} (Will never get hit as is)");
    }
    catch(OperationCanceledException)
    {
        _logger.LogInformation("!!!OperationCancelled!!!  Start clean up from here");
    }
    catch(Exception ex)
    {
        _logger.LogCritical(ex, $"Exception Caught:  {ex.GetType().FullName}");
    }

    _logger.LogInformation($"WORKER STOPPING at: {DateTimeOffset.Now}");
}

.NET Core 3.1 - Worker Service in linux container: Clean Shutdown

1 Answers