Using Spring Batch job for processing data fetched from Rest api

Viewed 23

There is a requirement where I need to read and process data fetched from a rest api let say restApi1 and write to different rest api let say restApi2 . For this I am using chunk oriented approach . But the issue is currently the restApi1 is not paginated . That endpoint returns a large number of data approximately 10000 .

So if my step failed then while restarting I have to read all the data again and process .
I can not start from where it failed .

Is this thought correct in relation to spring batch processing ?
Kindly suggest some possible approach .

ItemReader

public class MyItemReader extends ItemStreamSupport implements ItemReader<Data> {

private int curIndex = 0;

@Override
    public void open(ExecutionContext executionContext) {
        this.curIndex = 0;
    }

    @Override
    public void update(ExecutionContext executionContext) {
    }
}

ItemProcessor

public class MyItemProcessor extends ItemStreamSupport implements ItemProcessor<Data1, Data2> {
@Override
    public Data2 process(Data1 data1) throws Exception {
}
}

ItemWriter

public class MyItemWriter extends ItemStreamSupport implements ItemWriter<Data2> {
@Override
    public void write(List<? extends Data2> listOfData) throws Exception {
        
    }
}
1 Answers

You can download data to a file or a staging table in a tasklet step, then make your item reader read items from there.

In case of failure, the tasklet step should not be restarted, whereas your chunk-oriented step would resume from where it left off in the previous run from the file or staging table.

Related