How to pass machine input and Lambda function output to map states

Question

Current Setup:

I currently have a Step Functions state machine that kicks off a Task state (which calls a Lambda function), followed by a map state (which submits a job to Batch), defined as follows

State Machine Definition

(note: region and account id have been omitted and substituted for dummy variable ACCOUNT_INFO)

{
  "StartAt": "Populate EFS",
  "States": {
    "Populate EFS": {
      "Next": "MapState",
      "Type": "Task",
      "InputPath": "$",
      "ResultPath": "$.populate_efs_result",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": {
        "FunctionName": "arn:aws:lambda:{ACCOUNT_INFO}:function:PopulateEFSLambda",
        "Payload.$": "$"
      }
    },
    "MapState": {
      "Type": "Map",
      "End": true,
      "ResultPath": "$.metadata.run_info",
      "InputPath": "$",
      "Iterator": {
        "StartAt": "TaskState",
        "States": {
          "TaskState": {
            "Type": "Task",
            "End": true,
            "InputPath": "$",
            "ResultPath": null,
            "Resource": "arn:aws:states:::batch:submitJob.sync",
            "Parameters": {
              "JobDefinition": "arn:aws:batch:{ACCOUNT_INFO}:job-definition/BatchJobDefCfn:1",
              "JobName": "test",
              "JobQueue": "arn:aws:batch:{ACCOUNT_INFO}:job-queue/BatchQueue123",
              "ContainerOverrides": {
                "Command": [
                  "sh",
                  "-c",
                  "entrypoint.pl -i /NGS/${sequencer}/${run_id}/ -s ${sample_name}"
                ],
                "Environment": [
                  {
                    "Name": "run_id",
                    "Value.$": "$.run_id"
                  },
                  {
                    "Name": "sample_name",
                    "Value.$": "$.sample_name"
                  },
                  {
                    "Name": "sequencer",
                    "Value.$": "$.sequencer"
                  }
                ]
              }
            }
          }
        }
      }
    }
  }
}

State machine input

{
  "metadata": {
    "run_info": [
      {
        "sample_name": "SAMPLE_X",
        "sequencer": "Nextseq"
      },
      {
        "sample_name": "SAMPLE_Y",
        "sequencer": "Nextseq"
      },
      {
        "sample_name": "SAMPLE_Z",
        "sequencer": "Nextseq"
      }
    ]
  }
}

Lambda output (shortened for simplicity)

{"populate_efs_result": {
    "ExecutedVersion": "$LATEST",
    "Payload": "RUN_1"}

Expected Outcome:

The second step (MapState) needs information from the machine input (sample_name and sequencer), as well as what the Lambda function returns in populate_efs_result.Payload (run_id), therefore both need to be included in the event object for the Map state input. However, in my attempts so far, the input for the map state has been either the machine input or the Lambda output, not both.

I've tried changing the InputPath and ItemsPath parameters in the Map state definition and have also tried including the following in the Map state definition, but none of these methods work: Parameters: {"new_run_id.$": "$.populate_efs_result.Payload"}.

user_1569577 · Answer

A simple but not so elegant solution could be to move your lambda step within the Map state. The advantage would be that the response from the lambda would be in the context of the map state (this might be needed if your lambda response if specific to each iteration of the map state). The disadvantage is that your lambda function would need to be executed for each iteration of the map, while lambda function are fast and cheap it is still not a perfect solution.

Another approach would be to pass the execution input to that lambda and then extend the lambda to modify the run_info array to include the required data. Then pass this modified array to the map state as the InputPath

How to pass machine input and Lambda function output to map states

1 Answers