I'm trying to create an API which takes in a .zip file does some calculations on the file's data and returns a .zip file to a user with the same files plus one extra generated file. My idea was to use a stream and do calculations to zip file content on as data comes in and then pass it right out.
I have a controller method:
@RequestMapping(value = "/sign", consumes = "application/zip", produces = "application/zip", method = RequestMethod.POST)
public void calculate(HttpServletRequest request, HttpServletResponse response) throws Exception {
response.setStatus(HttpServletResponse.SC_OK);
response.addHeader("Content-Disposition", "attachment; filename=\"result.zip\"");
ZipOutputStream responseOutputStream = new ZipOutputStream(response.getOutputStream());
byte[] calculationResult = calculationService.doCalculations(request.getInputStream(), responseOutputStream);
//Write some more stuff to responseOutputStream based on calculations result
request.getInputStream().close();
responseOutputStream.close();
}
and a service method:
public byte[] doCalculations(InputStream zipFileStream, ZipOutputStream outputStream) throws Exception {
ByteArrayOutputStream calculationResultStream = new ByteArrayOutputStream();
ZipInputStream zipInputStream = new ZipInputStream(zipFileStream);
ZipEntry zipEntry = zipInputStream.getNextEntry();
while (zipEntry != null) {
//Output entry
ZipEntry zipOutEntry = new ZipEntry(zipEntry.getName());
outputStream.putNextEntry(zipOutEntry);
int bytesRead;
byte[] tempBuffer = new byte[8192 * 10];
while ((bytesRead = zipInputStream.read(tempBuffer)) != -1) {
outputStream.write(tempBuffer, 0, bytesRead);
}
//Do some quite light calculations
//calculationResultStream.write(byte[] calcRes);
zipEntry = zipInputStream.getNextEntry();
}
calculationResultStream.close();
return calculationResultStream.toByteArray();
}
The idea I had was to never write anything in memory to avoid being out of memory with large files. When I give it a .zip file with a single large (2GB) file, it processes it in ~25 seconds, but the problem arises when I give a .zip file containing 1000 small files (1-5MB in size each) then it runs for around 10 minutes and makes Postman crash.
Perhaps data is still cached somewhere and I'm approaching this incorrectly. How could I improve on this solution and why is it so slow with large amount of files and not with a few large files?