Getting cvs rows from aws s3.getObject.createReadStream() in node lambda

Viewed 329

I'm try to grab a csv file out of an s3 bucket using a node lambda and do database operations with each row. I'm able to log the stream itself, but don't get anything beyond that before my lambda closes.

Been shifting around trying different ways with different npm libraries but can't seem to ever get into any operations beyond getting the stream itself.

  const csv = require('fast-csv');
  
  class S3Service {
    constructor(s3=new AWS.S3()) {
      this.s3 = s3;
    }

    async _createReadStream(bucket, key) {
      console.log('getting a stream');
      return this.s3.getObject({ Bucket: bucket, Key: key }).createReadStream();
    }

    async readFileStreamRowByRow(bucket, key) {
      return new Promise(async (resolve, reject) => {
        const rows = [];
        console.log('inside readFileStreamRowByRow');
        const stream = await this._createReadStream(bucket, key);
        console.log('here is the stream', stream);
        stream.pipe(csv.parse({headers: true}))
          .on('error', error => console.error(error))
          .on('data', row => rows.push(row))
          .on('end', () => {
            console.log(rows);
            resolve(rows);
          })
      })
  }
1 Answers

To answer my own question, createReadStream() instantiates an instance whether or not S3 has successfully fetched an object or not. In my case, I had a VPC issue and the lambda was not even accessing the bucket and timing out later, but I was still logging a stream synchronously.

Related