Combining methods to TAR + LZ4 in a single run in Java

Question

I have two methods that separately TAR a set of files and then another that will compress it with LZ4. They both work fine, but I'm wondering if combining them together would be more efficient or save some time? Also I'm not really sure how I would combine them. Any suggestions would be useful. As you can see in my code below, I'm trying to have fine access to the data so I can give a good % complete to the user.

public static boolean createTarFile(List paths, Path output, int machineId)
{
    boolean success = false;
    boolean failure = false;

        try (OutputStream fOut = Files.newOutputStream(output, StandardOpenOption.APPEND);
                BufferedOutputStream buffOut = new BufferedOutputStream(fOut);
                TarArchiveOutputStream tOut = new TarArchiveOutputStream(buffOut))
        {
            float index = 1;
            for (Path path : paths)
            {

                TarArchiveEntry tarEntry = new TarArchiveEntry(
                        path.toFile(),
                        path.getFileName().toString());
                tarEntry.setSize(path.toFile().length());

                tOut.putArchiveEntry(tarEntry);

                // copy file to TarArchiveOutputStream
                Files.copy(path, tOut);

                tOut.closeArchiveEntry();
                tarPercentComplete = (index / (float) paths.size()) * 100;
                index++;
                if (abort)
                {
                    break;
                }

            }
            tOut.finish();
        }
        catch (Exception e)
        {
            LOG.error("Tarring file failed ", e);
            failure = true;
        }
    }

    return success;
}

/**
 * Zip a tar file using LZ4
 * 
 * @param fileToZip
 * @param outputFileName
 * @return
 */
public boolean zipFile(File fileToZip, File outputFileName)
{
    boolean success = false;
    boolean failure = false;

        try (FileOutputStream fos = new FileOutputStream(outputFileName);
                LZ4FrameOutputStream lz4fos = new LZ4FrameOutputStream(fos);)
        {
            try (FileInputStream fis = new FileInputStream(fileToZip))
            {
                byte[] buf = new byte[bufferSizeZip];
                int length;
                long count = 0;
                while ((length = fis.read(buf)) > 0)
                {
                    lz4fos.write(buf, 0, length);

                    if (count % 50 == 0)
                    {
                        zipPercentComplete = ((bufferSizeZip * count) / (float) fileToZip.length()) * 100;
                    }

                    count++;
                    if (abort)
                    {
                        break;
                    }
                }
            }
        }
        catch (Exception e)
        {
            LOG.error("Zipping file failed ", e);
            failure = true;
        }

    }

    return success;
}

user_5221149 · Answer

Just chain them.

try (OutputStream fOut = Files.newOutputStream(output, StandardOpenOption.APPEND);
     BufferedOutputStream buffOut = new BufferedOutputStream(fOut);
     LZ4FrameOutputStream lz4fos = new LZ4FrameOutputStream(buffOut);)
     TarArchiveOutputStream tOut = new TarArchiveOutputStream(lz4fos)) {

No need to name them all.

try (TarArchiveOutputStream tOut = new TarArchiveOutputStream(
        new LZ4FrameOutputStream(
           new BufferedOutputStream(
              Files.newOutputStream(output, StandardOpenOption.APPEND))))) {

user_646947 · Answer

Combining tar + lz4 in a single passs will definitely be more efficient, because it will avoid moving data to/from storage for a content which is inherently temporary.

On a shell (command line), one would do something like that :

tar cvf - DIR | lz4 > DIR.tar.lz4

which uses stdout / stdin as the intermediate interface (instead of storage file I/O).

However, as you are not using shell in your code, prefer following suggestions from @Andreas for a Java example. The main idea is the same, but the implementation is definitely different.

Combining methods to TAR + LZ4 in a single run in Java

2 Answers