How to restore your timer tasks & their timers which you created in memory after server restart (i.e. deployment)

user_2207316 0
Viewed 181

I am working on one application where we need to process some business logic before some specific time of the event's [it's application's entity] start time [field of an event entity which contains start time of an event]

My application picks up these events and creates timer task and assigns that timer task [user defined] to timer so java can execute at the given time [x mins/hours before the event's start time].

Now, I am facing issue that if server/spring boot service gets down so timer tasks and timers will be destroyed and I won't be able to process the business logic.

My Solution Approach (didn't work well):

To overcome to that issue , I have tried below logic,

  1. I am adding records in new table of eligible events and
  2. marking them "to be processed" before I create timer task
  3. once they are processed, we are marking them as "done".

Working with Single server instance: Now if server/service gets down, I will have records in new table with "to be processed" which I can pick up and process on server start up event. This will work for single server instance but we have multi server instances in our infrastructure.

Problem with multiple server instances: So main issue is how to handle this scenario in multiple server instances? How to make sure that only one server instance will pick up the record and will execute as there are chances that other server can also pick up the record and process it so there are chances of duplicate timer tasks of same event in multiple server instances and can create an issue which I need to avoid.

timertask spring-boot timer spring java
1 Answers

You can use the following "states" for your events that are stored in the table:

public enum EventState {
    READY, 
    TAKEN,
    DONE
}

When a new event is produced, you add it to the table and mark it as READY.

When a thread picks an event to process it, it will flag it as TAKEN then will start processing. Threads only pick READY events, no others.

When the thread finishes to process the event, it will flag it as DONE (or simply remove it from the table, that's up to you and your needs of keeping events that have already been processed).

In case the service crashes, when you restart it the main thread (before any parallel thread is started) will go through the table and check if there are threads that are in state TAKEN.

If so, it means that one thread had taken that event, but didn't have the time to finish it before the crash. Hence, all these TAKEN events will be re-marked as READY so that can be processed again by a new thread in this shot.

Of course, if you have multiple threads that read the table, you should put in place some lock mechanism that make sure that only one thread at a time can update the table.

Usually you can do that with synchronized method, for example:

public synchronized Event pickEvent() {
    //Search next READY event in the table
    //Flag it as TAKEN
    //Return it to the thread that should process it
}

The fact that the above method is synchronized guarantees you that only one thread per time can enter the method, which allows you not to worry about two threads coming at the same time and picking the same event.

Vishal made me notice you may have multiple JVMs reading the same table. If that's the case, synchronized will not work fine because it only synchronizes threads within the same JVM. In this case then, you will need to lock the table each time one thread gets in and unlock each time one thread gets out. You can do that by creating a small service that performs the read/write operations in the table, instead of reading/writing the table directly from the threads.

user_31111490
Related