I have written a small proof of concept Spring Boot / Batch application that uses JDBC to page through a very large database table (potentially billions of rows) and converts the rows read to XML before writing them out to an S3 bucket. Currently my PoC has a single job that commences on start up and will keep running until the table has been completely consumed. It runs against test data, which is only a few hundred rows. For my single job I am using:
JdbcPagingItemReader- A custom
ItemProcessorto convert rows to XML - A custom
ItemWriterto write XML to S3
I have a number of problems which I don't know how to address using Spring Batch. Having read the manual, nothing is jumping out at me as a solution for how I might progress. For the sake of argument lets say the table contains 1 billion rows of Person information and looks like this:
| pk | name | creation_date | payload_data |
|---|---|---|---|
| 1 | Jim | 2020-10-10 | Some |
| 2 | Bob | 2020-10-10 | Data |
| 3 | Jon | 2020-10-11 | In |
| 4 | Sue | 2020-10-14 | JSON |
- How do I ensure that if the Spring Boot container is killed, the job does not simply start again from the beginning? Processing these rows could potentially take over a week given the volume in the target table. I have considered changing the SQL query on the
JdbcPagingItemReaderso that it only reads a day at a time (adding aWHEREclause on creation date), but it's unclear how to:- Ensure that I pass increasing dates between jobs? There is something called a
JobParametersIncrementerwhich looks promising, but it's not clear how it works. - How do I ensure that the right starting parameter is given to the job if the system restarts? Do I somehow need to query the Spring Batch database at start up?
- Ensure that I pass increasing dates between jobs? There is something called a
- Related to the above is how do I track overall progress of the job? I can query the database I'm loading from to get number of rows, but I'm not sure how to get "current job parameters for the current job instance" so I can calculate how far along the application is in the processing.
If anyone has any design tips for the above, or has done something similar with Spring Batch, please feel free to add a comment.