I have a scenario where I have million records in employee staging table, I need to enrich the value and store it in employee final table. Now I am using chunk processing with chunk size of 10,000.
In processor where I have to do enriching I want to collect all employee-id for that chunk and do 1 API call per chunk and enrich value. Instead of million call for million records.
I am observing that reader I am using right now which is extending RepositoryItemReader is not returning List as I am using JPA I am using RepositoryItemReader. So processing is happening 1 time per item even after chunk size as 10,000.
Can we get the whole List from reader and do processing on it? Or is there any other approach, cause I cant really make 1 call per record.
public class EmployeeStagingReader extends RepositoryItemReader<EmployeeStaging>{
public EmployeeStagingReader(EmployeeStagingRepository repo){
super();
this.setRepository(repo);
this.setMethodName("findAll");
final Map<String,Sort.Direction> sorts = new HashMap<>();
sorts.put("ID",Sort.Direction.ASC)
this.setSort(sorts);
}
}
public class EmployeeProcessor implements ItemProcessor<List<EmployeeStaging>, List<EmployeeFinal>> {
//Want to Perform transformation of stagingemployee list of records and return employeefinal list of records
}
}
@Bean
public Step step1() {
return this.stepBuilderFactory.get("step1")
.<List<EmployeeStaging>, List<EmployeeFinal>>chunk(1000)
.reader(EmployeeStagingReader())
.processor(EmployeeProcessor())
.writer(EmployeeFinalWriter())
.build();
}