I am learning to create AWS Lambdas. I want to create a "chain": S3 -> 4 Chained Lambda()'s -> RDS. I can't get the first lambda to call the second

Viewed 77

I really tried everything. Surprisingly google has not many answers when it comes to this.

When a certain .csv file is uploaded to a S3 bucket I want to parse it and place the data into a RDS database.

My goal is to learn the lambda serverless technology, this is essentially an exercise. Thus, I over-engineered the hell out of it.

Here is how it goes:

  1. S3 Trigger when the .csv is uploaded -> call lambda (this part fully works)
  2. AAA_Thomas_DailyOverframeS3CsvToAnalytics_DownloadCsv downloads the csv from S3 and finishes with essentially the plaintext of the file. It is then supposed to pass it to the next lambda. The way I am trying to do this is by putting the second lambda as destination. The function works, but the second lambda is never called and I don't know why.
  3. AAA_Thomas_DailyOverframeS3CsvToAnalytics_ParseCsv gets the plaintext as input and returns a javascript object with the parsed data.
  4. AAA_Thomas_DailyOverframeS3CsvToAnalytics_DecryptRDSPass only connects to KMS, gets the encrcypted RDS password, and passes it along with the data it received as input to the last lambda.
  5. AAA_Thomas_DailyOverframeS3CsvToAnalytics_PutDataInRds then finally puts the data in RDS.

I created a custom VPC with custom subnets, route tables, gateways, peering connections, etc. I don't know if this is relevant but function 2. only has access to the s3 endpoint, 3. does not have any internet access whatsoever, 4. is the only one that has normal internet access (it's the only way to connect to KSM), and 5. only has access to the peered VPC which hosts the RDS.


This is the code of the first lambda:

// dependencies
const AWS = require('aws-sdk');
const util = require('util');
const s3 = new AWS.S3();
let region = process.env;


exports.handler = async (event, context, callback) =>
{
    var checkDates = process.env.CheckDates == "false" ? false : true;
    var ret = [];
    var checkFileDate = function(actualFileName)
    {
        if (!checkDates)
            return true;
            
        var d = new Date();
        var expectedFileName = 'Overframe_-_Analytics_by_Day_Device_' + d.getUTCFullYear() + '-' + (d.getUTCMonth().toString().length == 1 ? "0" + d.getUTCMonth() : d.getUTCMonth()) + '-' + (d.getUTCDate().toString().length == 1 ? "0" + d.getUTCDate() : d.getUTCDate());

        return expectedFileName == actualFileName.substr(0, expectedFileName.length);
    };

    for (var i = 0; i < event.Records.length; ++i)
    {
        var record = event.Records[i];
        
        try {
            if (record.s3.bucket.name != process.env.S3BucketName)
            {
                console.error('Unexpected notification, unknown bucket: ' + record.s3.bucket.name);
                continue;
            }
            
            if (!checkFileDate(record.s3.object.key))
            {
                console.error('Unexpected file, or date is not today\'s: ' + record.s3.object.key);
                continue;
            }

            const params = {
                Bucket: record.s3.bucket.name,
                Key: record.s3.object.key
            };
            var csvFile = await s3.getObject(params).promise();
            var allText = csvFile.Body.toString('utf-8');
            
            console.log('Loaded data:', {Bucket: params.Bucket, Filename: params.Key, Text: allText});
            
            ret.push(allText);
    
        } catch (error) {
            console.log("Couldn't download CSV from S3", error);
            return { statusCode: 500, body: error };
        }  
    }
    
    // I've been randomly trying different ways to return the data, none works. The data itself is correct , I checked with console.log()
    const response = {
        statusCode: 200,
        body: { "Records": ret }
    };
    
    return ret;
};

While this shows how the lambda was set up, especially its destination:

Screenshot of AWS Lambda web page

I haven't posted on Stackoverflow in 7 years. That's how desperate I am. Thanks for the help.

2 Answers

Rather than getting each Lambda to call the next one take a look at AWS managed service for state machines, step functions which can handle this workflow for you.

By providing input and outputs you can pass output to the next function, with retry logic built into it.

If you haven't much experience AWS has a tutorial on setting up a step function through chaining Lambdas.

By using this you also will not need to account for configuration issues such as Lambda timeouts. In addition it allows your code to be more modular which improves testing the individual functionality, whilst also isolating issues.

The execution roles of all Lambda functions, whose destinations include other Lambda functions, must have the lambda:InvokeFunction IAM permission in one of their attached IAM policies.

Here's a snippet from Lambda documentation:

To send events to a destination, your function needs additional permissions. Add a policy with the required permissions to your function's execution role. Each destination service requires a different permission, as follows:

  • Amazon SQS – sqs:SendMessage
  • Amazon SNS – sns:Publish
  • Lambda – lambda:InvokeFunction
  • EventBridge – events:PutEvents
Related