SpringBatch application periodically pulling data from DB

user_8735121 0
Viewed 384

I am working on a spring batch service that pulls data from a db on a schedule. (e.g. every day at 12pm) I am using JdbcPagingItemReader to read the data and a scheduler (@Scheduled provided by spring batch) to launch the job. The problem that I have now is: every time the job runs, it will just pull all the data from the beginning and not from the "last read" row. The data from the db is changing everyday(deleting old ones and adding new ones) and all I have is a timestamp column to track them.

Is there a way to "remember" the last row read from the last execution of the job and read data only later than that row?

spring-batch scheduled-tasks database
2 Answers

Since you need to pull data on a daily basis, and your records have a timestamp, then you can design your job instances to be based on a given date (ie using the date as an identifying job parameter). With this approach, you do not need to "remember" the last processed record. All you need to do is process records for a given date by using the correct SQL query. For example:

Job instance ID Date Job parameter SQL
1 2021-03-22 date=2021-03-22 Select c1, c2 from table where date = 2021-03-22
2 2021-03-23 date=2021-03-23 Select c1, c2 from table where date = 2021-03-23
... ... ... ...

With that in place, you can use any cursor-based or paging-based reader to process records of a given date. If a job instance fails, you can restart it without a risk to interfere with other job instances. The restart could be done even several days after the failure since the job instance will always process the same data set. Moreover, in case of failure and job restart, Spring Batch will reprocess records from the last check point in the previous (failed) run.

user_50193860

Just want to post an update to this question.

So in the end I created two more steps to achieve what I wanted to do initially. Since I don't have the privilege to modify the table where I read the data from, I couldn't use the "process indicator pattern" which involves having a column to mark if a record is processed or not. I created another table to store the last-read record's timestamp, and use it to update the sql query.

  1. step 0: a tasklet that reads the bookmark from a table, pass it in the job context
  2. step 1: a chunk step, get the bookmark from the context, use jdbcPagingItemReader to read the data
  3. step 2: a tasklet to update the bookmark

But doing this I have to be very cautious with the bookmark table. If I lose that I lose everything

user_87351210
Related