I am using Spring Batch Partitioning strategy to setup an ETL process (Copy data from one db to another), potentially few million records.
When I increase the input payload size to more than a million, the process fails with Out of Memory (Testing with 14GB RAM). After analyzing, I discovered that SpringBatch is creating significant byte[] and String objects, which are holding significant portion of memory. These objects were created by MapStepExecutionDao class, while trying to save the StepExecution for each partition. Below is the call flow:
org.springframework.util.SerializationUtils.serialize(Object) org.springframework.batch.core.repository.dao.MapStepExecutionDao.copy(StepExecution) org.springframework.batch.core.repository.dao.MapStepExecutionDao.saveStepExecution(StepExecution) org.springframework.batch.core.repository.dao.MapStepExecutionDao.saveStepExecutions(Collection) org.springframework.batch.core.repository.support.SimpleJobRepository.addAll(Collection)
In this case, the ExecutionContext within StepExecution has one large string (in format “abc, def, ………………………xyz”), I have to use this as an input to query data due to table structure. MapStepExecutionDao.copy serialize and then deserialize the StepExecution object to create a copy of it. Is there a way to SKIP the “serialization/deserialization” of StepExecution to get its copy, as it is creating additional byte[] and strings which are taking significant heap, and heap consumption increase (more than I expect) with input size.
private static StepExecution copy(StepExecution original) { return (StepExecution)SerializationUtils.deserialize(SerializationUtils.serialize(original)); }
Please let me know
- If there is a way to skip the “serialization/deserialization” of StepExecution ?
- Is there any configuration in SpringBatch that can avoid “serialization/deserialization”
- Is there a way to extend/override this behavior