Can We Process Whole Chunk Together in Spring Batch Processor

Viewed 460

I have a scenario where I have million records in employee staging table, I need to enrich the value and store it in employee final table. Now I am using chunk processing with chunk size of 10,000.

In processor where I have to do enriching I want to collect all employee-id for that chunk and do 1 API call per chunk and enrich value. Instead of million call for million records.

I am observing that reader I am using right now which is extending RepositoryItemReader is not returning List as I am using JPA I am using RepositoryItemReader. So processing is happening 1 time per item even after chunk size as 10,000.

Can we get the whole List from reader and do processing on it? Or is there any other approach, cause I cant really make 1 call per record.

public class EmployeeStagingReader extends RepositoryItemReader<EmployeeStaging>{
     public EmployeeStagingReader(EmployeeStagingRepository repo){
       super();
       this.setRepository(repo);
       this.setMethodName("findAll");
       final Map<String,Sort.Direction> sorts = new HashMap<>();
       sorts.put("ID",Sort.Direction.ASC)
       this.setSort(sorts);
  }
}


public class EmployeeProcessor implements ItemProcessor<List<EmployeeStaging>, List<EmployeeFinal>> {
  
        //Want to Perform transformation  of stagingemployee list of records and return employeefinal list of records

    }
}


@Bean
public Step step1() {
    return this.stepBuilderFactory.get("step1")
                .<List<EmployeeStaging>, List<EmployeeFinal>>chunk(1000)
                .reader(EmployeeStagingReader())
                .processor(EmployeeProcessor())
                .writer(EmployeeFinalWriter())
                .build();
}
0 Answers
Related