AWS Lambda unzip from S3 to S3

Question

I'm trying to write an Lambda function that unzips zip files in one S3 directory and extract into another. I had this working in Python but nobody else in my group likes Python so I'm converting it to Node.js which I'm not very good at.

I'm trying to use the unzipper package and I'm able to get a list of files in the zip file using unzipper.Open.S3, but I can't figure out how to stream the files in the zip file into S3.

The meat of the code looks like

  const directory = await unzipper.Open.s3(s3,{Bucket: bucket, Key: zip_file});

  directory.files.forEach(file => {
    console.log("file name = " + file.path + ", type = " + file.type)
    const key = dir[0] + "/output/" + file.path;
    const params = { Bucket: bucket, Key: key };
    const { writeStream, promise } = uploadStream(params)
    file.stream().pipe(writeStream);
    promise.then(() => {
      console.log('upload completed successfully');
    }).catch((err) => {
      console.log('upload failed.', err.message);
    });
  });

const uploadStream = ({ Bucket, Key }) => {
  const pass = new stream.PassThrough();
  return {
    writeStream: pass,
    promise: s3.upload({ Bucket, Key, Body: pass }).promise()
  };
}

I get the console.log for each file, but neither of the logs in promise.then and .catch comes out and no new files appear in S3.

Paul_Tomblin_3333 · Answer

Never mind, I found this code that works better:

exports.handler = async (event) =>      {
    const params = {
    Key: zip_directory + "/" + zip_file,
    Bucket: input_bucket
  };

  const zip = s3
      .getObject(params)
      .createReadStream()
      .pipe(unzipper.Parse({ forceStream: true }));

  const promises = [];

  let num = 0;

  for await (const e of zip) {
    const entry = e;

    const fileName = entry.path;
    const type = entry.type;
    if (type === 'File') {
      const uploadParams = {
        Bucket: output_bucket,
        Key: output_directory + fileName,
        Body: entry,
      };

      promises.push(s3.upload(uploadParams).promise());
      num++;
    } else {
      entry.autodrain();
    }
  }

  await Promise.all(promises);

};

AWS Lambda unzip from S3 to S3

1 Answers