LZ4 and Zstd for Java

user_13185778 0
Viewed 4272

Is there any Best Java Compression Library available for LZ4 and ZStd. I have tried with apache commons ( which is zstd-jni implementation)

            String fileURL = TestFileUtil.getFileURL(TestFileCategory.SMALL);
            String outputFileName = TestFileUtil.BASE_DIR+"/zstd-"+(Math.random()*10)+".x";
            System.out.println(Paths.get(fileURL));
            printFileInfo(fileURL);
            StopWatch watch = new StopWatch();          
            InputStream in = Files.newInputStream(Paths.get(fileURL));
            OutputStream fout = Files.newOutputStream(Paths.get(outputFileName));
            BufferedOutputStream out = new BufferedOutputStream(fout);
            ZstdCompressorOutputStream zOut = new ZstdCompressorOutputStream(out);
            int buffersize = 1024*4;
            watch.mark();
            final byte[] buffer = new byte[buffersize];
            int n = 0;
            while (-1 != (n = in.read(buffer))) {
                zOut.write(buffer, 0, n);
            }
            zOut.close();
            in.close();

But this code doesnt work it throws 

Exception in thread "main" java.lang.NoClassDefFoundError: com/github/luben/zstd/ZstdOutputStream
    at org.apache.commons.compress.compressors.zstandard.ZstdCompressorOutputStream.<init>(ZstdCompressorOutputStream.java:83)
    at com.zoho.test.testzstd.main(testzstd.java:28)
Caused by: java.lang.ClassNotFoundException: com.github.luben.zstd.ZstdOutputStream
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 2 more

and for LZ4 I just replace ZStdCompressorOutputStream with 

FramedLZ4CompressorOutputStream lzOut = new FramedLZ4CompressorOutputStream(out);

But it tooks almost 2hrs (not yet completed ) to compress 2.4GB (csv) file. Is there anything wrong with the code? or anyother suggestions?

zstd lz4 lossless-compression file
1 Answers

For zstd, Java has @luben's zstd-jni : https://github.com/luben/zstd-jni

If you prefer Java native, @martint has a complete port in Java (though with less features) at : https://github.com/airlift/aircompressor/tree/master/src/main/java/io/airlift/compress/zstd

For LZ4, a common binding is : https://github.com/lz4/lz4-java , though if you prefer an implementation which is compatible with the lz4 CLI, the Apache Commons version is preferable.

These libraries generally cannot be swapped by just changing one invocation name. Refer to their documentations or examples in order to understand how they work.

user_6469470
Related