How to run a glue crawler using the AWS javascript SDK?

This is a snippet from what I have deployed. The crawler is created based off of the params1. Then once created runs the crawler. Hope that helps. The code originated from

https://aws.amazon.com/blogs/big-data/build-and-automate-a-serverless-data-lake-using-an-aws-glue-trigger-for-the-data-catalog-and-etl-jobs/

var params = {
            DatabaseInput: {
                Name: dbName,
                Description: dbDecription,
            }
        };

glue.createDatabase(params, function(err, data) {
                var params1 = {
                    DatabaseName: "database",
                    Name: "crawler",
                    Description: 'crawler test',
                    Role: crawlerRoleArn,
                    Targets: {
                        S3Targets: [{ Path: bucket }]
                    },
                };
                glue.createCrawler(params1, function(err1, data1) {
                    startCrawler(crawlerName, function(err2,data2){
                        if(err2) callback(err2)
                        else callback(null,data2)
                    })
                });
        });

Also trying to do this, trigger a crawler from lambda (not create and trigger). I went and dug through the documentation.

1. Google "aws-sdk glue", top result looks good.

2. We know we can use createCrawler as @pkarfs showed above. Looking at the method summary I can see `getCrawler

https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Glue.html#getCrawler-property

There's a sample on the page but here's what I went with. I constructed this by using F12 to inspect the types on the startCrawler function.

  /**
   * Starts a crawl using the specified crawler, regardless of what is scheduled. If the crawler is already running, returns a CrawlerRunningException.
   */
  startCrawler(params: Glue.Types.StartCrawlerRequest, callback?: (err: AWSError, data: Glue.Types.StartCrawlerResponse) => void): Request<Glue.Types.StartCrawlerResponse, AWSError>;

As you can see the params is an interface of type StartCrawlerRequest. This only needs a name. If you are creating your crawler in CloudFormation you can pass the crawler name into the lambda function as an environment variable.

You will also need to give your lambda function permission to start the crawler. I googled "glue api permissions reference" and also guessed it would be something like resource:ActionName turns out it's glue:StartCrawler as can be seen here.

import { Handler } from "aws-lambda";
import { Glue } from "aws-sdk";
import "source-map-support/register";

export const handler: Handler = async () => {
  const glue = new Glue();

  const params: Glue.GetCrawlerRequest = {
    Name: "CRAWLER_NAME"
  };

  try {
    const data = await glue.startCrawler(params).promise();

    // eslint-disable-next-line no-console
    console.log({ data });
  } catch (err) {
    console.error(`Error getting crawler instance: ${err}`);
  }

  console.log({ helloWorld: "yo" });
};

I'm using serverless.

  myCrawlerTrigger:
    handler: ./src/lambdas/myfunction.handler
    events:
      - s3:
          bucket: MY_BUCKET
          event: s3:ObjectCreated:Put
          rules:
            # TODO switch to json.gz after compressing the content
            - prefix: bucket_folder_name/
            - suffix: .json
          existing: true # my bucket already exists

https://www.serverless.com/framework/docs/providers/aws/events/s3/

But how did you create the glue crawler and database?

See this