ClientWebsocket in BackgroundService(IHostedService) stops working silently

Viewed 392

I have a problem where my client websocket implementation within my ASP NET Core web application hosted in an Azure as a App Service stops working after an irregular amount of time.

The implementation is based on a BackgroundService that is run on startup.

public class MyService : BackgroundService
{
    public IServiceProvider _services { get; }

    public MyService(IServiceProvider services)
    {
        _services = services;
    }

    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        var fortnoxKeys = new List<string>();
        
        using (var scope = _services.CreateScope())
        {
            var userManager = scope.ServiceProvider.GetService<UserManager<ApplicationUser>>();
            if (userManager == null)
            {
                throw e;
            }

            fortnoxKeys = FortnoxHelper.getAllFortnoxKeys(userManager);
        }

        var client = new FortnoxWebSocketClient("MY-KEY");
        await client.Connect();

        await client.AddTenant(fortnoxKeys);
        var addTenantResponse = await client.Receive();
        
        await client.AddTopic(WebSocketTopic.Orders);
        var addTopicResponse = await client.Receive();

        await client.Subscribe();
        var subscribeResponse = await client.Receive();

        while (!stoppingToken.IsCancellationRequested)
        {
            var Listen = Task.Run(async () =>
            {
                while (true)
                {
                    var response = await client.Receive();

                    if (response.Type == WebSocketResponseType.EventResponse)
                    {
                        if (response.EventType == WebSocketEventType.OrderCreated)
                        {

                          try
                          {
                              // Do work
                          }
                          catch (Exception e)
                          {
                              continue;
                          }

                          // Do more work
                        }
                    }
                    else if (response.Type == WebSocketResponseType.CommandResponse)
                    {
                    }
                }
            });

            var Update = Task.Run(async () =>
            {
                while (true)
                {
                    // Do work and decide which tenants to remove and which to add
                    await client.RemoveTenant(tenantId);
                    // ...
                    await client.AddTenant(newFortnoxKeys);
                }
            });

            Task.WaitAll(Update, Listen);
        }
    }
}

This service is injected as a singleton within Startup.cs the reason behind it being a singleton is because I want only a single instance of this websocket client to be run.

services.AddSingleton<IHostedService, MyService>();

The idea behind the implementation is for this to run forever. Listening to messages and occasionaly update the websocket within the Update task. However this only works for an irregular amount of time. One time it works for 1 day and next time it works for 20 days before it stops recieving messages.

The FortnoxWebSocketClient is a wrapper for the ClientWebsocket Class and the source code for it can be found here

I have tried upgrading the App Service plan to Standard because I thought it might be the resource recycling in Azure that was the problem but this was unsuccessfull. I have also set the "Always on" flag in Azure to On. In our Azure logs we can see the following message after a certain amount of downtime:

Heartbeat took longer than "00:00:01" at "03/11/2021 04:09:49 +00:00"

0 Answers
Related