How to return output from AWS Glue jobs back to the calling Step Function workflow?

user_1451834 0
Viewed 6172

AWS Step Functions allow calling AWS Glue jobs, as described here: https://docs.aws.amazon.com/step-functions/latest/dg/connect-glue.html

I want to run the job and (after saving the results to S3) return some metadata produced during the job (like row count or number of filtered rows) back to the Step function flow.

We can pass parameters from Step functions to the Glue job like this:

              "RunGlueJob": {
                "Type": "Task",
                "Resource": "arn:aws:states:::glue:startJobRun.sync",
                "Parameters": {
                  "JobName": "MyJobName",
                  "Arguments": {
                    "--param1.$": "$.param1",
                    "--param2.$": "$.param2"
                  }
                },
                "Next": "NextState"
              },

But how can the Glue job return output back to the Step Function workflow? I tried just returning a String from the main() function inside the (Scala) Glue job, but it doesn't show up in JSON returned to the step function flow:

{
      "AllocatedCapacity": 3,
      "Arguments": {
        "--param1.$": "$.param1",
        "--param2.$": "$.param2"
      },
      "Attempt": 0,
      "CompletedOn": 1570114802442,
      "ExecutionTime": 39,
      "GlueVersion": "0.9",
      "Id": "jr_some_id",
      "JobName": "MyJobName",
      "JobRunState": "SUCCEEDED",
      "LastModifiedOn": 1570114802442,
      "LogGroupName": "/aws-glue/jobs",
      "MaxCapacity": 3,
      "PredecessorRuns": [],
      "StartedOn": 1570114746138,
      "Timeout": 2880
    }

I cannot find any documentation on this, so it might be that this is simply not possible. However, returning values from Lambdas works just fine and shows up normally inside the Step function workflow.

aws-step-functions aws-glue amazon-web-services
2 Answers

You can't return anything from glue job at this stage. By definition, AWS glue is expected to work on huge amount of data and hence it is expected that output will also be huge amount of data.

You may write result to dynamodb or s3 or any other storage and access it using lambda in next step in AWS step functions

user_76133400

AWS Glue is a fully managed extract, transform, and load (ETL) service and is nothing else but a wrapper on top of Spark. When you execute a task in Glue ETL, it actually submits a spark job and executes the your code. In general, spark is supposed to load the data from hadoop supported filesystems like HDFS/S3 (hadoop treats s3 as a filesystem), applies the transformation and writes the processed data back to disk (which is the output of a spark program).

In case you really want to return some value from your spark program, you could create a listener on some port and publish the data you want to return as response to that port. However, given the restricted environment of glue, implementing any such work around would be a tedious job.

user_51649360
Related