Asynchronous code works independently, but not together. Not getting any exceptions, just timing out

Viewed 86

I am writing a lambda function to add hosts to a SQS queue for a rolling restart. The code I have written works individually, but not together. Even when I hard code values in the constructor. This doesn't appear to be a memory/CPU. I tried running the function with 1GB of memory, even though it only uses about 80MB. The average execution time for the individual functions is about 0.5 seconds (shouldn't take more than about 1.5 seconds to execute in total). I did trying running this function with a 30 second timeout, but it still timed out.

I work behind a corporate proxy, and have to hand jam the code. I don't have an IDE or intellisense on my internet facing network. There may be typos here, but not in the actual code. I have omitted my module imports and variable declarations to save time. It isn't relevant to the issue at hand.

EDIT: I added the module imports and variable declarations to the first example to hopefully alleviate some confusion.

Here are just a few things I have tried. This does not work (timing out):

// Custom lambda layer
const { marklogic, aws } = require('nodejs-layer-lib'); 
const { HOSTS, DOMAIN, PORT, USERNAME, PASSWORD, RESTART_QUEUE_NAME } = process.env;

const params = [
  'format=json'
];

const options = {
  port: PORT,
  params: params,
  httpOptions: {
    headers: {
       'Authorization': `Basic ${Buffer.from(`${USERNAME}:${PASSWORD}`).toString('base64')}`
    },
    method: 'GET'
  }
};

const taskServers = (HOSTS.split(',') || []).map(host => {
  const _host = host.split(':');

  return {
    id: _host[0],
    name: `http://${_host[1].toLowerCase()}.${DOMAIN}`
  };
});

exports.handler = async () => {
  let hosts, queueUrl, addToQueueResults;

  try {
    hosts = (await marklogic.hosts.getHosts(taskServers, options) || []);
  } catch (e) { console.error('hosts', e); }

  try {
    queueUrl = await aws.sqs.getQueueUrlByName(RESTART_QUEUE_NAME);
  } catch (e) { console.error('queueUrl ', e); }

  try {
    addToQueueResults = await aws.sqs.addMessages(queueURL, hosts);
  } catch (e) { console.error('addToQueueResults ', e); }

  return {
    status: 200,
    body: addToQueueResults
  };
}

This does not work (timing out):

// Modules imports and variable declarations here...

exports.handler = async () => {
  const hosts = (await marklogic.hosts.getHosts(taskServers, options) || []);
  const queueUrl = await aws.sqs.getQueueUrlByName(RESTART_QUEUE_NAME);
  const addToQueueResults = await aws.sqs.addMessages(queueURL, hosts);

  return {
    status: 200,
    body: addToQueueResults
  };
}

This does not work (timing out):

// Modules imports and variable declarations here...

exports.handler = async () => {
  const hosts = (await marklogic.hosts.getHosts(taskServers, options) || []);
  const queueUrl = await aws.sqs.getQueueUrlByName('my-queue-name');
  const addToQueueResults = await aws.sqs.addMessages('http://queueurl.com', ['anything', 'in', 'here']); // Doesn't even need the queueUrl or hosts anymore

  return {
    status: 200,
    body: addToQueueResults
  };
}

This works. It will return the host objects I am expecting in the response:

// Modules imports and variable declarations here...

exports.handler = async () => {
  const hosts = (await marklogic.hosts.getHosts(taskServers, options) || []);

  return {
    status: 200,
    body: hosts 
  };
}

This works. It will get the queue url, then add messages to my SQS queue and return the SQS response:

// Modules imports and variable declarations here...

exports.handler = async () => {
  const queueUrl = await aws.sqs.getQueueUrlByName(RESTART_QUEUE_NAME);
  const addToQueueResults = await aws.sqs.addMessages(queueUrl , ['anything', 'in', 'here']);

  return {
    status: 200,
    body: addToQueueResults
  };
}

I tried implementing the Async handler in AWS Lambda function handler in Node.js and reviewed many AWS Lambda execution troubleshooting documents. The marklogic management API runs on port 8002 by default and I think the aws-sdk module uses http/https (80/443), so I don't think the ports are getting tied up.

What am I missing here?

EDIT 2: This has something to do with how promises are handled with AWS Lambda. I cannot find much information about this. Even following the instructions in AWS Lambda function handler in Node.js for "Async Handlers" I cannot get this to work. It works perfectly fine locally with or without my custom lambda layer.

Node.js runtime: 12.x (I didn't mention this before)

This also doesn't work (timing out):

// Modules imports and variable declarations here...

exports.handler = async function (event) {
  const promise = function () {
      return new Promise(async function (resolve, reject) {
        try {
          const hosts = await marklogic.hosts.getHosts(taskServers, options) || [];
          const queueUrl = await aws.sqs.getQueueUrlByName(RESTART_QUEUE_NAME);
          const addToQueueResults = await aws.sqs.addMessages(queueUrl, hosts); 
          resolve({
            status: 200,
            body: addToQueueResults
          });
        } catch (error) {
           reject({
             status: 500,
             error: error
           });
        }
      });
  };

  return promise(); // Throws error without constructor despite the AWS doc example
}

Unless someone AWS Lambda genius has ran into a similar issue before with Node.js, I am just going to convert it into 2 lambda functions and use Step Functions to process them.

1 Answers

There was a typo in queueUrl (I imagine not that, but worth a try!)

Please run:

// Custom lambda layer
const { marklogic, aws } = require('nodejs-layer-lib');

const { HOSTS, DOMAIN, PORT, USERNAME, PASSWORD, RESTART_QUEUE_NAME } = process.env;

const params = [
    'format=json'
];

const options = {
    port: PORT,
    params,
    httpOptions: {
        headers: {
            Authorization: `Basic ${Buffer.from(`${USERNAME}:${PASSWORD}`).toString('base64')}`
        },
        method: 'GET'
    }
};

const taskServers = (HOSTS.split(',') || []).map(host => {
    const _host = host.split(':');

    return {
        id: _host[0],
        name: `http://${_host[1].toLowerCase()}.${DOMAIN}`
    };
});

exports.handler = async () => {
    let hosts, queueUrl, addToQueueResults;

    try {
        hosts = (await marklogic.hosts.getHosts(taskServers, options) || []);
    } catch (e) { console.error('hosts', e); }

    try {
        queueUrl = await aws.sqs.getQueueUrlByName(RESTART_QUEUE_NAME);
    } catch (e) { console.error('queueUrl ', e); }

    try {
        addToQueueResults = await aws.sqs.addMessages(queueUrl, hosts);
    } catch (e) { console.error('addToQueueResults ', e); }

    return {
        status: 200,
        body: JSON.stringify(addToQueueResults)
    };
};

// keeping the same format.. ^^

If no luck - what is on my mind, is as aws-sdk is included out of the box in lambda.. it's not customary to require it extraneously via a layer and although it may not look to be imported at top level by marklogic, it may be bundled deep within marklogic, then when you import AWS and change config (in the layer) it overwrites it

Let's find out..:

Step 1:

So, this you say should work.. if we ignore the AWS import, and just import marklogic?

// Custom lambda layer
// const { marklogic, aws } = require('nodejs-layer-lib'); // ignoring AWS for now
const { marklogic } = require('nodejs-layer-lib');

const { HOSTS, DOMAIN, PORT, USERNAME, PASSWORD, RESTART_QUEUE_NAME } = process.env;

const params = [
    'format=json'
];

const options = {
    port: PORT,
    params,
    httpOptions: {
        headers: {
            Authorization: `Basic ${Buffer.from(`${USERNAME}:${PASSWORD}`).toString('base64')}`
        },
        method: 'GET'
    }
};

const taskServers = (HOSTS.split(',') || []).map(host => {
    const _host = host.split(':');

    return {
        id: _host[0],
        name: `http://${_host[1].toLowerCase()}.${DOMAIN}`
    };
});

exports.handler = async () => {
    // let hosts, queueUrl, addToQueueResults;
    let hosts;

    try {
        hosts = (await marklogic.hosts.getHosts(taskServers, options) || []);
        console.log('hosts => ', hosts);

        // queueUrl = await aws.sqs.getQueueUrlByName(RESTART_QUEUE_NAME);

        // addToQueueResults = await aws.sqs.addMessages(queueUrl, hosts);

        return {
            status: 200,
            body: JSON.stringify(hosts)
        };
    } catch (error) {
        console.log('error => ', error);

        throw error;
    }
};

Ok, so if that works..:

Step 2 (Please set the region for SQS and also hard code in the queueUrl):

// Custom lambda layer
// const { marklogic, aws } = require('nodejs-layer-lib');
const { marklogic } = require('nodejs-layer-lib');

const AWS = require('aws-sdk');

AWS.config.update({ region: 'eu-west-1' }); // Please set region accordingly
const sqs = new AWS.SQS({ apiVersion: '2012-11-05' });

const { HOSTS, DOMAIN, PORT, USERNAME, PASSWORD, RESTART_QUEUE_NAME } = process.env;

const params = [
    'format=json'
];

const options = {
    port: PORT,
    params,
    httpOptions: {
        headers: {
            Authorization: `Basic ${Buffer.from(`${USERNAME}:${PASSWORD}`).toString('base64')}`
        },
        method: 'GET'
    }
};

const taskServers = (HOSTS.split(',') || []).map(host => {
    const _host = host.split(':');

    return {
        id: _host[0],
        name: `http://${_host[1].toLowerCase()}.${DOMAIN}`
    };
});

exports.handler = async () => {
    let hosts, addToQueueResults;

    try {
        hosts = (await marklogic.hosts.getHosts(taskServers, options) || []);
        console.log('hosts => ', hosts);

        const queueUrl = 'Please hard code the queueUrl for now';

        const sqsParams = {
            MessageBody: hosts,
            QueueUrl: queueUrl
        };

        addToQueueResults = await sqs.sendMessage(sqsParams).promise();

        console.log('addToQueueResults => ', addToQueueResults);

        return {
            status: 200,
            body: JSON.stringify(addToQueueResults)
        };
    } catch (error) {
        console.log('error => ', error);

        throw error;
    }
};

IF.. that doesn't work.. then Step 3.. move the require of marklogic to below the require of AWS and setting the region in this last example.. (so any deeply nested marklogic AWS logic we're unaware of now overwrites your AWS require..) re-run it.. fingers crossed :-)

Related