Split/join a binary file into multiple parts without loading file into memory?

Nicholas DiPiazza 0
Viewed 52

In Java, how do you split a binary file into multiple parts while only loading a small portion of the File into memory at one time?

So I have a file FullFile that is large. I need to upload it to cloud storage but it's so large that it often times out.

I can make this problem less likely if I split the file and upload in chunks.

So I need to split FullFile into files of chunk size MaxChunkSize.

  • List<File> fileSplit(File fullFile, int maxChunkSize)
  • File fileJoin(List<File> splitFiles)

Most code snippets around require the file to be text. But in my case the files are compressed binary.

What would be the best way to implement these methods?

java
1 Answers

Below is the full answer:

  • The maxChunkSize represents the size in bytes of a file chunk.

  • In the example below I read a 5mb zip file and split it into five 1MB chunks and later join them back using the fileJoin function.

  • The method stageLocally stages the files locally but you can modify it to work with any cloud storage. (Better to abstract this out so you can switch between multiple storage implementations)

  • You can tweak maxChunkSize based on the amount of data you want to store inmemory at a given time

  • The IOutils.copy() methods is from the commons library, here is the maven link. You can also use Files.copy() in liue of it. The Files.copy() methods comes from the java.nio package, so you don't have to add an external dependency to use it.

I have ommitted the exception handling for brevity.

public static void main(String[] args) throws IOException {

    File input = new File(_5_MB_FILE_PATH);
    File outPut = fileJoin(split(input, 1_024_000));

    System.out.println(IOUtils.contentEquals(Files.newInputStream(input.toPath()), Files.newInputStream(outPut.toPath())));

}

public static List<File> split(File largeFile, int maxChunkSize) throws IOException {
    InputStream in = Files.newInputStream(largeFile.toPath());
    List<File> list = new ArrayList<>();
    final byte[] buffer = new byte[maxChunkSize];
    int dataRead = in.read(buffer);
    while (dataRead > -1) {
        list.add(stageLocally(buffer, dataRead));
        dataRead = in.read(buffer);
    }
    return list;
}

private static File stageLocally(byte[] buffer, int length) throws IOException {
    File outPutFile = File.createTempFile("temp-", "split", new File(TEMP_DIRECTORY));
    FileOutputStream fos = new FileOutputStream(outPutFile);
    fos.write(buffer, 0, length);
    fos.close();
    return outPutFile;
}


public static File fileJoin(List<File> list) throws IOException {
    File outPutFile = File.createTempFile("temp-", "unsplit", new File(TEMP_DIRECTORY));
    FileOutputStream fileOutputStream = new FileOutputStream(outPutFile);
    for (File file : list) {
        InputStream in = Files.newInputStream(file.toPath());
        IOUtils.copy(in, fileOutputStream);
        in.close();
    }
    fileOutputStream.close();
    return outPutFile;
}

Let me know if this helps.

Malkeith Singh0
Related