I have a question regarding the handling of multiple jobs which are triggered by endpoints on multiple .Net Core and ASP-Net applications.
Currently we are managing this jobs with a scheduled windows task which is starting an executable. The jobs are defined in a database table and are basically entries for which endpoint is to call and when. Our Executable pulls the table in a thirty minutes interval and checks which endpoint is to call.
The Problem is that we are waiting for their completion in a sequential order, means that each endpoint is triggered after another. So we have a lack of concurrency here. Moreover it is not possible to control (e.g. stop a running task) and monitor the triggered endpoints.
We already looked into frameworks like hangfire and the basic task and concurrency concepts of C#, but we only find solutions for managing jobs and long running tasks in a single application.
So our question is basically, how can we manage our jobs in a centralized way, when the jobs are distributed over many different web server applications? And how can we get concurrency and control over the triggered tasks?
We liked the idea of defining our jobs in a central database table. But triggering the Executable every thirty minutes does not seem to be the proper way of handling multiple background jobs.
Also endpoints are not suitable for executing long running tasks, so how can we manage concurrent jobs in "different assemblies".
Which is the best approach to tackle this problem?