Worker service stops working unexpectedly

user_5338928 0
Viewed 1759

I have .NET Core 3+ worker service that checks "some stuff" every 10 seconds. At one point, it "randomly" stopped doing that and I am not sure why. So far it happened twice and there are no exception logs or anything like it, so I can only assume that I should add a try-catch inside ExecuteAsync, but my question is: Are there any known issues that could cause a similar behavior where the worker just stops executing?

And yes, there was no cancellation requested (else, I suppose it would log the message "Sender is stopping").

public class Worker : BackgroundService
{
    private readonly ILogger<Worker> _logger;

    public Worker(ILogger<Worker> logger)
    {
        _logger = logger;
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        _logger.LogInformation("Sender is starting");
        stoppingToken.Register(() => _logger.LogInformation("Sender is stopping"));

        while (!stoppingToken.IsCancellationRequested)
        {
            await Task.Delay(10000, stoppingToken);

            _logger.LogDebug("Worker running at: {time}", DateTimeOffset.Now);

            if (!stoppingToken.IsCancellationRequested)
                await SendPendingMessages();
        }
    }

    private async Task SendPendingMessages()
    {
        var list = ...

        foreach (var item in list)
        {
            try
            {
                // Do stuff as expected
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, ex.Message);
            }
        }
    }
}
service-worker worker .net-core c#
3 Answers

So far it happened twice and there are no exception logs or anything like it, so I can only assume that I should add a try-catch inside ExecuteAsync, but my question is: Are there any known issues that could cause a similar behavior where the worker just stops executing?

It definitely sounds like an exception to me. By default, the BackgroundService implementation will ignore any exceptions thrown from ExecuteAsync. If an exception is raised from ExecuteAsync, it is ignored and the service just stops running.

I always recommend a top-level try/catch with logging, so that you're at least aware that this happens:

protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
  _logger.LogInformation("Sender is starting");
  try
  {
    while (!stoppingToken.IsCancellationRequested)
    {
      await Task.Delay(10000, stoppingToken);
      _logger.LogDebug("Worker running at: {time}", DateTimeOffset.Now);
      if (!stoppingToken.IsCancellationRequested)
        await SendPendingMessages();
    }
  }
  catch (Exception ex) when (stoppingToken.IsCancellationRequested)
  {
    _logger.LogInformation("Sender is stopping");
  }
  catch (Exception ex) when (!stoppingToken.IsCancellationRequested)
  {
    _logger.LogError(ex, "Sender had error");
  }
}

On a related note, if this is what you would consider a "critical" background service, then you would also want to stop the application host when this service stops:

private readonly ILogger<Worker> _logger;
private readonly IHostApplicationLifetime _hostApplicationLifetime;

public Worker(ILogger<Worker> logger, IHostApplicationLifetime hostApplicationLifetime)
{
  _logger = logger;
  _hostApplicationLifetime = hostApplicationLifetime;
}

protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
  _logger.LogInformation("Sender is starting");
  try
  {
    while (!stoppingToken.IsCancellationRequested)
    {
      await Task.Delay(10000, stoppingToken);
      _logger.LogDebug("Worker running at: {time}", DateTimeOffset.Now);
      if (!stoppingToken.IsCancellationRequested)
        await SendPendingMessages();
    }
  }
  catch (Exception ex) when (stoppingToken.IsCancellationRequested)
  {
    _logger.LogInformation("Sender is stopping");
  }
  catch (Exception ex) when (!stoppingToken.IsCancellationRequested)
  {
    _logger.LogError(ex, "Sender had error");
  }
  finally
  {
    _hostApplicationLifetime.StopApplication();
  }
}
user_2636930

Be aware that the line

await Task.Delay(10000, stoppingToken);

will throw OperationCancelledException when you trigger your cancellation token, it is unhandled.

Also make sure the LogError() LogDebug() have a catch all try block and cannot throw themselves, that would also break your code.

Can't see anything else that's problematic.

user_63145990
user_84989350
Related