Zip file as an input and output for REST API

Viewed 384

I'm trying to create an API which takes in a .zip file does some calculations on the file's data and returns a .zip file to a user with the same files plus one extra generated file. My idea was to use a stream and do calculations to zip file content on as data comes in and then pass it right out.

I have a controller method:

@RequestMapping(value = "/sign", consumes = "application/zip", produces = "application/zip", method = RequestMethod.POST)
public void calculate(HttpServletRequest request, HttpServletResponse response) throws Exception {

    response.setStatus(HttpServletResponse.SC_OK);
    response.addHeader("Content-Disposition", "attachment; filename=\"result.zip\"");

    ZipOutputStream responseOutputStream = new ZipOutputStream(response.getOutputStream());

    byte[] calculationResult = calculationService.doCalculations(request.getInputStream(), responseOutputStream);

    //Write some more stuff to responseOutputStream based on calculations result

    request.getInputStream().close();
    responseOutputStream.close();
}

and a service method:

  public byte[] doCalculations(InputStream zipFileStream, ZipOutputStream outputStream) throws Exception {

    ByteArrayOutputStream calculationResultStream = new ByteArrayOutputStream();

    ZipInputStream zipInputStream = new ZipInputStream(zipFileStream);

    ZipEntry zipEntry = zipInputStream.getNextEntry();

    while (zipEntry != null) {
        //Output entry
        ZipEntry zipOutEntry = new ZipEntry(zipEntry.getName());
        outputStream.putNextEntry(zipOutEntry);

        int bytesRead;
        byte[] tempBuffer = new byte[8192 * 10];

        while ((bytesRead = zipInputStream.read(tempBuffer)) != -1) {
            outputStream.write(tempBuffer, 0, bytesRead);
        }

        //Do some quite light calculations
        //calculationResultStream.write(byte[] calcRes);

        zipEntry = zipInputStream.getNextEntry();
    }

    calculationResultStream.close();
    return calculationResultStream.toByteArray();
}

The idea I had was to never write anything in memory to avoid being out of memory with large files. When I give it a .zip file with a single large (2GB) file, it processes it in ~25 seconds, but the problem arises when I give a .zip file containing 1000 small files (1-5MB in size each) then it runs for around 10 minutes and makes Postman crash.

Perhaps data is still cached somewhere and I'm approaching this incorrectly. How could I improve on this solution and why is it so slow with large amount of files and not with a few large files?

0 Answers
Related