Retrieving the progress of getObject (aws-sdk)

user_4556172 0
Viewed 511

I'm using node.js with the aws-sdk (for S3).... When I am downloading a huge file from s3, how can I regularly retrieve the progress of the download so that the front-end can show a progress bar? Currently I am using getObject. (https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#getObject-property)

The code to download the file works. Here's a snippet of my code...

 return await new Promise((resolve, reject) => {
  this.s3.getObject(params, (error, data) => {
    if (error) {
      reject(error);
    } else {
      resolve(data.Body);
    }
  });

I'm just not sure how to hook into the progress as it's downloading. Thanks in advance for any insight!

amazon-web-services amazon-s3
2 Answers

You can utilize S3 byte-range fetching which allows the fetching of small parts of a file in S3. This capability then allows us to fetch large objects by dividing the file download into multiple parts which brings the following advantages:

  • Part download failure does not require full re-downloading of the file.
  • Download pause/resume capability.
  • Download progress tracking
  • Retry packets that failed or interrupted by network issues
  • Sniff headers located in the first few bytes of the file if we just need to get metadata from the files.

You can split the file download by your size of choice (I propose 1-4mb at a time) and download the parts chunk by chunk, when each of the get object promises complete, you can trace how many have completed. A good start is by looking at the AWS documentation.

user_9834760

STREAMING OPTION

Another option is to use a stream and track the amount of bytes received:

    const { ContentLength: contentLength } = await s3.headObject(params).promise();

    const rs = s3.getObject(s3Params).createReadStream();

    let progress = 0;
    rs.on('data', function (chunk) {
      // Advance your progress by chunk.length
      progress += chunk.length;
      console.log(`Progress: ${progress / contentLength}%`);
    });

    // ... pipe to write stream
user_48937530
Related