Auto-distribution Queue System for Workers

user_19331305 0
Viewed 59

Server

I am working on a server to manage several worker instances through TCP Sockets. However, I am having an issue distributing the tasks evenly among the worker instances.

Worker

The worker can have a set number of slots (threads) for tasks, and the amount of workers fluctuates from time to time. I want to be able to add and remove a worker on-demand. The tasks should be redistributed (so that each worker has tasks proportional to their capacity) in the event of the addition/deletion of workers. I also want to be able to add and remove tasks on-demand.

Example with Table View

Worker 1 (Slot Capacity of 2) Worker 2 (Slot Capacity of 2) Worker 3 (Slot Capacity of 8)
task1 task3 task5
task2 task4 task6
task7
task8
task9
task10
task11
task12

Are there any existing libraries that can help with this data structure? If not, what's my best route for making my own data structure like this?

data-structures java
1 Answers

First I will state my assumptions:

It sounds like the "workers" are individual processes on an operating system, and not a thread within a single program/process.

Why would the amount of workers fluctuate from time to time? Due to network partitions, crashes, or because you dynamically wish to spin them down due to cost?

Now I will state a potential answer:

You should use the "server" you mention as a type of "commander" who is responsible for confirming that the workers are still working on their tasks. They can send heartbeats to the commander that say that they are still alive and connected and working on a specific task. Similarly, the commander should be able to send messages to the worker with a link/path on where to find the next chunk of data to work on and a place for the worker to output the work.

In the event that you kill or a worker dies in the middle of the task, the commander should be responsible for checking if a taskId is complete or not, and re-assign accordingly to a node with no pending tasks.

You could write this all yourself if you need the performance of TCP, or something like Zookeeper or Kafka might help with the communication patterns.

Additionally, classes like CompleteableFuture in Java would be nice for non-blocking async communication between the commander and workers.

user_186105100
Related