.Net Core Hosted Services in a Load Balanced Environment

Question

We are developing a Web API using .Net Core. To perform background tasks we have used Hosted Services.

System has been hosted in AWS Beantalk Environment with the Load Balancer. So based on the load Beanstalk creates/remove new instances of the system.

Our problem is, Since background services also runs inside the API, When load balancer increases the instances, number of background services also get increased and there is a possibility to execute same task multiple times. Ideally there should be only one instance of background services.

One way to tackle this is to stop executing background services when in a load balanced environment and have a dedicated non-load balanced single instance environment for background services only.

That is a bit ugly solution. So,

1) Is there a better solution for this?

2) Is there a way to identify the primary instance while in a load balanced environment? If so I can conditionally register Hosted services.

Any help is really appreciated.

Thanks

user_7054483 · Answer

I am facing the same scenario and thinking of a way to implement a custom service architecture that can run normally on all of the instance but to take advantage of pub/sub broker and distributed memory service so those small services will contact each other and coordinate what's to be done. It's complicated to develop yes but a very robust solution IMO.

user_4550944 · Answer

You'll "have to" use a distributed "lock" system. You'll have to use, for example, a distributed memory cache who put a lock when someone (a node of your cluster) is working on background. If another node is trying to do the same job, he'll be locked by the first lock if the work isn't done yet. What i mean, if all your nodes doesn't have a "sync handler" you can't handle this kind of situation. It could be SQL app lock, distributed memory cache or other things ..

.Net Core Hosted Services in a Load Balanced Environment

2 Answers