aws glue job dependency in step function

user_6288820 0
Viewed 7799

I've created 2 glue jobs (gluejob1, gluejob2).

I want create a dependency as gluejob2 should run only after gluejob1 is completed.

To orchestrate this, I created a step function with below definition:

 {
  "gluejob1": {
    "Type": "Task",
    "Resource": "gluejob1.Arn",
    "Comment": "Glue job1.",
    "Next": "gluejob2"
  },

  "gluejob2": {
    "Type": "Task",
    "Resource": "gluejob2.Arn",
    "Comment": "TGlue job2.",
    "Next": "Gluejob2 Finished Loading"
  },
  "Gluejob2 Finished Loading": {
    "Type": "Pass",
    "Result": "",
    "End": true
  }
}

When I execute this step function, state function calls it a success the moment it triggers the Gluejob1 and moves on to trigger gluejob2.

I'm wondering if there is a possibility to run gluejob2 only after gluejob1 completes.

aws-step-functions aws-glue amazon-web-services
3 Answers

You can invoke Glue job from StepFunction synchronously so that it will wait for job completion:

{
  "StartAt": "gluejob1",
  "States": {
    "gluejob1": {
      "Type": "Task",
      "Resource": "arn:aws:states:::glue:startJobRun.sync",
      "Parameters": {
        "JobName.$": "ETLJobName1"
      },
      "Next": "gluejob2"
    },
    "gluejob2": {
      "Type": "Task",
      "Resource": "arn:aws:states:::glue:startJobRun.sync",
      "Parameters": {
        "JobName.$": "ETLJobName2"
      },
      "Next": "Gluejob2 Finished Loading"
    },
    "Gluejob2 Finished Loading": {
      "Type": "Pass",
      "Result": "",
      "End": true
    }
}
user_17915100
  1. We need to first make sure our IAM Step Function Role has the ability to trigger our glue job to run as well as get a "call back" from the aws glue job when the job has been completed or failed. The policy for your step function should include this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "glue:StartJobRun",
                "glue:GetJobRun",
                "glue:GetJobRuns",
                "glue:BatchStopJobRun"
            ],
            "Resource": "*"
        }
    ]
}
2. Now we can add the "StartJobRun" Step to trigger our Glue job. start glue job

  1. We need to enable "wait for job to complete" which appears in the visual workflow editor. This will make ensure that our step function will wait for this task to be completed before going to the next one. If this is not enabled it will just trigger and move immediately to your next glue job.

step function screenshot

I made a step-by-step tutorial walk-through on how to configure a step function to run glue jobs synchronously and how to configure the IAM policy correctly for the step function role so the step function will not have the hanging error that others have experienced:

https://youtu.be/-zm-1egM3hY

user_46948200

Why not use triggers in glue to handle dependencies?

user_76133400
Related