My project has one main node and one work node, which the work node excutes some job fetched from the main node(most jobs are time-sensitive like send an email after three hours).
We expect the work node to be stateless, it register to main node and fetch some jobs to do, when a job done, it send a finish signal. The main node will check work node's health every once in a while, if work node was died, it set the jobs to unfetched.
The point is we choose HTTP to connect main and worker, so the main node get infomation only when work node send a KEEPALIVE request. It makes some confuse in disaster recovery and status synchronization.
Wanted to know if it's good practice to do that and what would be the best way to do that?
Thanks in advance.