AWS Lambda function invoked from another Lambda function runs stale request in queue

Viewed 150

I have a Lambda function (A) that calls another Lambda function (B) and returns while the other one is still executing. Everything works perfectly locally (using sls offline), but when I deploy it, the following behavior occurs:

  1. I call A for the first time. I see on Cloudwatch that A runs successfully and returns, but nothing on B, which apparently doesn't run.
  2. I call A another time. Cloudwatch now shows that both A and B run, but the logs show that the invocation of B had request parameters corresponding to the first run of A.

Ensuing runs of A result in runs of B with request params from the previous run of A (e.g. B always lags behind by one and never runs the current invocation like it is supposed to, almost like each call of A pushes out the previous B invocation from a queue of size 1)

Some additional info:

  • No VPC's are involved (if that makes a difference?)
  • I am using sls

Anyone have any idea why this is happening? I know that AWS async invocations go into a queue, but I'm pretty sure they should just run ASAP if there's nothing in the queue.

Here's the code:

module.exports.A = async (event, context) => {
    context.callbackWaitsForEmptyEventLoop = false;

    try {
        ... some code ...

        await connectToDatabase();

        ... some code ...

        const newReq = req;
        newReq['newSessionId'] = savedSession._id.toString();
        const params = {
            FunctionName: config.apiName + '-' + config.apiVersion + '-B',
            InvocationType: 'Event',
            Payload: JSON.stringify(newReq)
        };

        lambda.invoke(params, (err, res) => {
            if (err) {
                console.log(err);
            } else {
                console.log(res);
            }
        });

        return {
            statusCode: 200,
            headers: {
                'Access-Control-Allow-Origin' : '*',
                'Access-Control-Allow-Credentials' : true 
            },
            body: JSON.stringify(savedSession)
        }
    } catch (err) {
        console.log(err);
        return {
            statusCode: err.statusCode || 500,
            headers: { 'Content-Type': 'text/plain' },
            body: 'Could not create session.'
        }
    }
}

module.exports.B = async (event, context) => {
    context.callbackWaitsForEmptyEventLoop = false;

    try {
        ... some code ....

        // Connect to Atlas
        await connectToDatabase();
        
        ... some code ...

        for (let i = 0; i < applicants.length; i++) {
            ... some code ...

            const savedResume = await DownloadResume(resumeId, accessToken, fileName, oauthSrcs.GSCRIPT);
            conversionPromises.push(ConvertAndExportToS3(fileName, savedResume, sessionKey));
        }

        await Promise.all(conversionPromises);

        ... some code ...

        return {
            statusCode: 200,
            headers: {
                'Access-Control-Allow-Origin' : '*',
                'Access-Control-Allow-Credentials' : true 
            },
            body: JSON.stringify('Converted all resumes successfully!')
        }
    } catch (err) {
        console.log(err);
        return {
            statusCode: err.statusCode || 500,
            headers: { 'Content-Type': 'text/plain' },
            body: 'Could not create session.'
        }
    }
}
1 Answers

I see you are returning from A without waiting for lambda.invoke to finish. I think you should.

To clarify, you are indeed invoking an asynchronous lamdda (meaning you don't have to wait for the child lambda to finish), but the invocation itself is synchronous. Not sure what would happen if you returns A in the middle of B's invocation.

Also, did you wait long enough so you are sure that the 2nd run of A triggers anything ? It could be that a cold start on B makes it look that way if you don't wait long enough on first call

Related