What is the most efficient way to display images from a ZIP archive on an offline desktop app?

user_9718056 0
Viewed 1394

I have an offline Electron + React desktop app which uses no server to store data. Here is the situation:

An user can create some text data and add any number of images to them. When they save the data, a ZIP with the following structure is created:

myFile.zip
  ├ main.json // Text data
  └ images
     ├ 00.jpg
     ├ 01.jpg
     ├ 02.jpg
     └ ...

If they want to edit the data, they can open the saved ZIP with the app.

I use ADM-ZIP to read the ZIP content. Once open, I send the JSON data to Redux since I need them for the interface display. The images, however, are not displayed on the component that read the archive and can be on multiple "pages" of the app (so different components). I display them with an Image component that takes a name props and return an img tag. My problem here is how to get their src from the archive.

So here is my question: what could be the most efficient way to get the data of the images contained in the ZIP given the above situation?

What I am currently doing:

When the user open their file, I extract the ZIP in a temp folder and get the images paths from there. The temp folder is only deleted when the user close the file and not when they close the app (I don't want to change this behaviour).

// When opening the archive
const archive = new AdmZip('myFiles.zip');
archive.extractAllTo(tempFolderPath, true);

// Image component
const imageSrc = `${tempFolderPath}/images/${this.props.name}`;

Problem: if the ZIP contains a lot of images, the user must have enough space on their disk to get them extracted properly. Since the temp folder is not necessarily deleted when the app is closed, it means that the disk space will be taken until they close the file.

What I tried:

  1. Storing the ZIP data in the memory

    I tried to save the result of the opened ZIP in a props and then to get my images data from it when I need them:

    // When opening the archive
const archive = new AdmZip('myFile.zip');
this.props.saveArchive(archive); // Save in redux

// Image component
const imageData = this.props.archive.readFile(this.props.name);
const imageSrc = URL.createObjectURL(new Blob([imageData], {type: "image/jpg"}));

    With this process, the images loading time is acceptable.

    Problem: with a large archive, storing the archive data in the memory might be bad for the performances (I think?) and I guess there is a limit on how much I can store there.

  2. Opening the ZIP again with ADM-ZIP every time I have to display an image

    // Image component
const archive = new AdmZip('myFile.zip');
const imageData = archive.readFile(this.props.name);
const imageSrc = URL.createObjectURL(new Blob([imageData], {type: "image/jpg"}));

    Problem: bad performances, extremely slow when I have multiple images on the same page.

  3. Storing the images Buffer/Blob in IndexedDB

    Problem: it's still stored on the disk and the size is much bigger than the extracted files.

  4. Using a ZIP that doesn't compress the data

    Problem: I didn't see any difference compared to a compressed ZIP.

What I considered:

Instead of a ZIP, I tried to find a file type that could act as a non-compressed archive and be read as a directory but I couldn't find anything like that. I also tried to create a custom file with vanilla Node.js but I'm afraid that I don't know the File System API enough to do what I want.

I'm out of ideas so any suggestions are welcome. Maybe I didn't see the most obvious way to do... Or maybe that what I'm currently doing is not that bad and I am overthinking the problem.

electron reactjs node.js zip javascript
2 Answers

What do you mean by "most efficient" is not exactly very clear. I'll make some assumptions and respond ahead according them.

Cache Solution

Basically what you have already done. It's pretty efficient loading it all at once (extracting to a temporary folder) because whenever you need to reuse something, the most spendable task doesn't have to be done again. It's a common practice of some heavy applications to load assets/modules/etc. at their startup.

Obs¹: Since you are considering the lack of disk space as a problem, if you want to stick with this is desirable that you handle this case programmatically and give the user some alert, since having little free space on storage is critical.

TL;DR - Spendable at first, but faster later for asset reutilization.

Lazy Loading

Another very common concept which consists in basically "load as you need, only what you need". This is efficient because this approach assures your application only loads the minimum needed to run and then load things as demanded by the user.

Obs¹: It looks pretty much on what you have done at your try number 2.

TL;DR - Faster at startup, slower during runtime.

"Smart" Loading

This is not a real name, but it describes satisfactorily what i mean. Basicaly, focus on understanding the goal of your project and mix both of previous solutions according to your context, so you can achieve the best overall performance in your application, reducing trade offs of each approach.

Ex:

  • Lazy loading images per view/page and keep a in-memory cache with limited size

  • Loading images in background while the user navigates

Now, regardless of your final decision, the follow considerations should not be ignored:

  1. Memory will always be faster to write/read than disk
  2. Partially unzipping (setting specific files) is possible in many packages (including ADM-ZIP) and is always faster than unzipping everything, specially if the ZIP file is huge.
  3. Using IndexedDB or a custom file-based database like SQLite offers overall good result for a huge number of files and a "smart" approach, since querying and organizing data is easier through those.
  4. Always keep in mind the reason of every application design choice, the best you understand why you are doing something, the better your final result will be.
  5. Topic 4 being said, in my opinion, in this case you really overthought a little, but that's not a bad thing and it's really admirable to have this kind of concern about how can you do things on the best possible way, I do this often even if it's not necessary and it's good for self improvement.

Well, I wrote a lot :P

I would be very pleased if all of this help you somehow.

Good luck!

TL;DR - There is not a closed answer based on your question, it depends a lot on the context of your application; some of your tries are already pretty good, but putting some effort into understanding the usability context to handle image loading "smartly" would surely award you.

user_122325090

The issue of using ZIP

Technically streaming the content would be the most efficient way but streaming zipped content from inside a ZIP is hard and inefficient by design since the data is fragmented and needs to parse the whole file to get the central directory

A much better format for your specific use case is 7zip it allows you to stream out single file and does not require you to read out the whole file to get the information about this single files.

The cost of using 7zip

You will have to add a 7zip binary to your program, luckily it's very small ~2.5MB.

How to use the 7zip

const { spawn } = require('child_process')
// Get all images from the zip or adjust to glob for the needed ones
// x, means extract while keeping the full path
// (optional instead of x) e, means extract flat
// Any additional parameter after the archive is a filter
let images = spawn('path/to/7zipExecutable', ['x', 'path/to/7zArchive', '*.png'])

Additional consideration

Either you use the Buffer data and display them inside the HTML via Base64 or you locate them into a cache folder that gets cleared as part of the extraction process. Depending on granular you want to do this.

Keep in mind that you should use appropriate folder for that under windows this would be %localappdata%/yourappname/cache under Linux and MacOS I would place this in a ~/.yourappname/cache

Conclusion

  • Streaming data is the most efficient (lowest memory/data footprint)
  • By using a native executable you can compress/decompress much faster then via pure js
  • the compression/decompression happens in another process avoiding your application from beeing blocked in the execution and rendering

ASAR as Alternative

Since you mentioned you considered to use another format that is not compressed you should give the asar file format a try it is a build in format in electron that can be accessed by regular filepaths via FS no extraction etc.

For example you have a ~/.myapp/save0001.asar which contains a img folder with multiple images in it, you would access that simply by ~/.myapp/save0001/img/x.png

user_51150090
Related