Specify Header and Footer for all file created by MultiResourceItemWriter when writing in multiples files -spring batch

user_7652300 0
Viewed 1679

I have a batch that read from data base using JdbcPagingItemReader, process each record from the database in a java class then write it to a file using FlatFileItemWriter. It appends also a header and footer for this file using FlatFileFooterCallback and FlatFileHeaderCallback

The Job work well and it gives as output, a single JSON file in this format :

{"informations":[
{
 "name" : "xxx",
 "adress" : "xxx"
  //a very complex json object (1000 lines)
},
{
   "name" : "xxx",
 "adress" : "xxx"
  //a very complex json object (1000 lines )
},

// Many objects
]}

Kindly note that the header is just this :

{"informations":[

And the footer is just

]}

now the file is too big and I want to split it to multiple file using MultiResourceItemWriter Each file has at maximum 1000 row read from the database.

So I configured the step and I used the first FlatFileItemWriter above and this work well. As result, I have many files that have 1000 records from a data base but without the header and the footer {"informations":[ and ]}

All files generated by MultiResourceItemWriter have this format :

{
 "name" : "xxx",
 "adress" : "xxx"
  a very complex json object (1000 lines)
},
{
   "name" : "xxx",
 "adress" : "xxx"
  a very complex json object (1000 lines )
},
{
// many objects
}

How could I add the header and the footer for all files created by MultiResourceItemWriter When writing.

I found an answer that say that we can't combine MultiResourceItemWriter with FlatFileItemWriter that have FlatFileFooterCallback and FlatFileHeaderCallback with spring batch version < 2.1 .

Stream closed exception when combining MultiResourceItemWriter and FlatFileItemWriter with footer callback and stack overflow post stack-overflow-post-about-my-problem

I have this exception java.lang.IllegalStateException: JsonWriter is closed when trying to wrap FlatFileItemWriter with FlatFileHeaderCallback in my final MultiResourceItemWriter writer.

So is there a way to specify a header and footer for all file created by MultiResourceItemWriter when writing in a new file (splitting to multiple file) ? Is there a way to define a template of the file in which the MultiResourceItemWriter write in?

If no, could you please guide me to do so?

I would like to have in final, all file that created by MultiResourceItemWriter have at the beginning the {"informations":[ and at the end ]} , all the file content is wrapped in the json array informations.

writer spring-batch spring json java
1 Answers

How could I add the header and the footer for all files created by MultiResourceItemWriter When writing.

You need to set the header/footer callbacks on your delegate writer. Here is a quick example:

import java.io.IOException;
import java.io.Writer;
import java.util.Arrays;

import org.springframework.batch.core.Job;
import org.springframework.batch.core.JobParameters;
import org.springframework.batch.core.Step;
import org.springframework.batch.core.configuration.annotation.EnableBatchProcessing;
import org.springframework.batch.core.configuration.annotation.JobBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepBuilderFactory;
import org.springframework.batch.core.launch.JobLauncher;
import org.springframework.batch.item.ItemReader;
import org.springframework.batch.item.ItemWriter;
import org.springframework.batch.item.file.FlatFileFooterCallback;
import org.springframework.batch.item.file.FlatFileHeaderCallback;
import org.springframework.batch.item.file.FlatFileItemWriter;
import org.springframework.batch.item.file.ResourceSuffixCreator;
import org.springframework.batch.item.file.builder.FlatFileItemWriterBuilder;
import org.springframework.batch.item.file.builder.MultiResourceItemWriterBuilder;
import org.springframework.batch.item.file.transform.PassThroughLineAggregator;
import org.springframework.batch.item.support.ListItemReader;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.ApplicationContext;
import org.springframework.context.annotation.AnnotationConfigApplicationContext;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.io.FileSystemResource;

@Configuration
@EnableBatchProcessing
public class MyJob {

    @Bean
    public ItemReader<Integer> itemReader() {
        return new ListItemReader<>(Arrays.asList(1, 2, 3, 4, 5, 6, 7, 8, 9, 10));
    }

    @Bean
    public ItemWriter<Integer> itemWriter() {
        FlatFileItemWriter<Integer> flatFileItemWriter = new FlatFileItemWriterBuilder<Integer>()
                .lineAggregator(new PassThroughLineAggregator<>())
                .name("itemsWriter")
                .headerCallback(writer -> writer.write("header"))
                .footerCallback(writer -> writer.write("footer"))
                .build();

        return new MultiResourceItemWriterBuilder<Integer>()
                .delegate(flatFileItemWriter)
                .resource(new FileSystemResource("items"))
                .itemCountLimitPerResource(5)
                .resourceSuffixCreator(index -> "-" + index + ".txt")
                .name("multiResourcesWriter")
                .build();
    }

    @Bean
    public Job job(JobBuilderFactory jobs, StepBuilderFactory steps) {
        return jobs.get("job")
                .start(steps.get("step")
                        .<Integer, Integer>chunk(5)
                        .reader(itemReader())
                        .writer(itemWriter())
                        .build())
                .build();
    }

    public static void main(String[] args) throws Exception {
        ApplicationContext context = new AnnotationConfigApplicationContext(MyJob.class);
        JobLauncher jobLauncher = context.getBean(JobLauncher.class);
        Job job = context.getBean(Job.class);
        jobLauncher.run(job, new JobParameters());
    }

}

This generates two files items-1.txt and items-2.txt with the following content:

header
1
2
3
4
5
footer

and

header
6
7
8
9
10
footer

I use spring batch version 3.0.10.RELEASE

This version is not maintained anymore, so I encourage you to upgrade to the latest and greatest v4.3.1. The example above uses Spring Batch v4.3.1 and works as expected.

user_50193860
Related