I am developing a batch application using (Spring boot, java, and Spring batch) for which I need to read data from different locations. Below is my use case:
- Multiple paths such as C://Temp//M1, C://Temp//M2 , both locations can contain identical files with same data such as C://Temp//M1//File1.txt, C://Temp//M2//File1.txt, and C://Temp//M1//File2.txt, C://Temp//M2//File2.txt
- At first, I need to merge them in memory if an identical file exists at both locations before starting batch after removing duplicates and pass the merged in-memory data as an argument to the reader.
I have designed batch using multiresourceitemreader which reads flat files and processes them but not able to achieve in-memory merging and duplicate removal from multiple files. So may you please have a look and suggest me a way how can I achieve this?