Unzip .gz with zlib & async/await (w/o using streams)

user_7475615 0
Viewed 1381

Since zlib has been added to node.js I'd like to ask a question about unzipping .gz with async/await style, w/o of using streams, one by one.

In the code below I am using fs-extra instead of standard fs & typescript (instead of js), but as for the answer, it doesn't matter will it have js or ts code.

import fs from 'fs-extra';
import path from "path";
import zlib from 'zlib';

(async () => {
  try {
    //folder which is full of .gz files.
    const dir = path.join(__dirname, '..', '..', 'folder');
    const files: string[] = await fs.readdir(dir);

    for (const file of files) {
      //read file one by one
      
      const
        file_content = fs.createReadStream(`${dir}/${file}`),
        write_stream = fs.createWriteStream(`${dir}/${file.slice(0, -3)}`,),
        unzip = zlib.createGunzip();

      file_content.pipe(unzip).pipe(write_stream);
    }
  } catch (e) {
    console.error(e)
  }
})()

As for now, I have this code, based on streams, which is working, but in various StackOverflow answers, I haven't found any example with async/await, only this one, but it also uses streams I guess.

So does it even possible?

//inside async function
const read_file = await fs.readFile(`${dir}/${file}`)
const unzip = await zlib.unzip(read_file);
//write output of unzip to file or console

I understand that this task will block the main thread. It's ok for me since I write a simple day schedule script.

node.js zlib javascript
1 Answers

Seems I have figure it out, but I am still not hundred percent sure about it, here is example of full IIFE:


(async () => {
  try {
    //folder which is full of .gz files.
    const dir = path.join(__dirname, '..', '..', 'folder');
    const files: string[] = await fs.readdir(dir);

    //parallel run
    await Promise.all(files.map(async (file: string, i: number) => {
      
      //let make sure, that we have only .gz files in our scope
      if (file.match(/gz$/g)) {
        const
          buffer = await fs.readFile(`${dir}/${file}`),
          //using .toString() is a must, if you want to receive readble data, instead of Buffer
          data = await zlib.unzipSync(buffer , { finishFlush: zlib.constants.Z_SYNC_FLUSH }).toString(),
          //from here, you can write data to a new file, or parse it.
          json = JSON.parse(data);

        console.log(json)
      }
    }))
  } catch (e) {
    console.error(e)
  } finally {
    process.exit(0)
  }
})()

If you have many files in one directory, I guess you could use await Promise.all(files.map => fn()) to run this task in parallel. Also, in my case, I required to parse JSON, so remember about some nuances of JSON.parse.

user_74756150
Related