In Node.js Create read stream from S3 object or web URL, and PassThrough the read stream to multiple pipes

user_1244597 0
Viewed 1468

In a Node.js12.x based Lambda function, I am attempting to

  1. Download object from S3
  2. Download file from web if object not exists in S3
  3. PassThrough the downloaded data regardless of how its arrived to multiple write streams

I tried following

var request = require('request');

const readStream = ({ Bucket, Key }) => {

  s3.getObjectMetadata({ Bucket, Key })
    .promise().then(() => {
      return s3.getObject().createReadStream();
    })
    .catch(error => {
      if (error.statusCode === 404) {
        return request.get('http://example.com/' + Key);
      }
    });
};

readStream({ ... })
  .pipe(sharp().resize(w, h).toFormat('png'))
  .pipe(writeStream);

Above works if the object available in s3, but the catch block doesn't work.

Do I need await or a promise on request.get?

I also tried following with no luck

http.get('http://example.com/' + Key, function(response) { 
    return response;
});
node.js amazon-web-services amazon-s3
1 Answers

First we need to create a promise that checks if the object exists in the bucket

const isObjectExists = (params) => {
  return s3.headObject(params)
           .promise()
           .then(
             () => true,
             (error) => {
               if (err.statusCode === 404) {
                 return false;
               }
               throw error;
             }
           )}

const awsParams = { Bucket: ..., Key: s3ObjectKey };
const exists = await isObjectExists(awsParams);

Next, we need to create a read stream from either s3 or web URL.

Using the deprecated request package, since I am not able to get things to work using http.

let readStream = null;
if (exists) {
    readStream = s3.getObject(awsParams).createReadStream();
} else {
    readStream = request.get('http://example.com/' + s3ObjectKey);
}

Pass the just created readStream through to process and finally to the upload stream

const pass = new PassThrough();

readStream.
  .pipe(sharp().resize(w, h).toFormat('png'))
  .pipe(s3.upload({
    Body: pass,
    Bucket,
    ContentType: 'image/png',
    s3ObjectKey + '-resized'
  }).promise());

const uploadedData = await pass;

Add few prerequisites

import awS3 from 'aws-sdk/clients/s3';
import { PassThrough } from 'stream'
import * as request from 'request';

const s3 = new awS3({signatureVersion: 'v4'});
user_12445970
Related