Multithreaded Unzipping In Java

user_1317823 0
Viewed 3662

So, I'm trying to do read-only access of a zip file in Java, decompressing in a multithreaded manner, because my standard simple singly-threaded solution of ZipFile/ZipEntry using the enumeration and inputstreams and what-not results in it taking about five full seconds just to decompress into memory a 50-meg zipfile which takes one second AT MOST for my disk to read without decompressing.

However, the entire Java zip library is synchronized to an incredibly-obnoxious degree, no doubt because it's all abstracted for reading/writing/etc. in the same code instead of having nice efficient non-synchronized read-only code.

I've looked at third-party Java libraries, and all of them are either massive VFS libraries that are worse than using an elephant gun to shoot a fly, or else the only reason they have a performance benefit is they multithread to the extent that most of the threads are blocking on disk IO anyway.

All I want to do is pull a zipfile into a byte[], fork some threads, and work on it. There's no reason any synchronization would be needed in any way for anything, because each of the unzipped files I use separately in memory without interaction.

Why must this be so difficult?

inflate zlib multithreading java
3 Answers

As you noticed, all the methods in ZipFile are synchronized. But that only stops multiple threads from running at the same time across different ZipFile instances, opened for the same exact zipfile on disk.

If you want multiple threads to read from the same zipfile in a scalable way, you must open one ZipFile instance per thread. That way, the per-thread lock in the ZipFile methods does not block all but one thread from reading from the zipfile at one time. It also means that when each thread closes the ZipFile after they're done reading, they close their own instance, not the shared instance, so you don't get an exception on the second and subsequent close.

Protip: if you really care about speed, you can get more performance by reading all the ZipEntry objects from the first ZipFile instance, and sharing them with all threads, to avoid duplicating work in reading ZipEntry objects for each thread separately. A ZipEntry object is not tied to a specific ZipFile instance per se, ZipEntry just records metadata that will work with any ZipFile object representing the same zipfile that the ZipEntry came from.

user_39509820
Related