AWS CDK - StepFunction - Modify input before passing to the next step

Viewed 3407

I have a 3 step state machine for a step function.

InputStep -> ExecuteSparkJob -> OutputLambda

ExecuteSparkJob is a glue task. Since it cannot pass its output to the step function, it writes it output to an S3 bucket. OutputLambda reads it from there and passes it on to the step function.

The idea of InputStep is simply to define a common S3 URI that the following steps can use.

Below is the code I have for the Input Step.


        const op1 = Data.stringAt("$.op1");
        const op2 = Data.stringAt("$.op2");

        const inputTask = new Pass(this, "Input Step", {
            result: Result.fromString(this.getURI(op1, op2)),
            resultPath: "$.s3path"
        });

    getURI(op1: string, op2: string): string {
        return op1.concat("/").concat(op2).concat("/").concat("response");
    }


However, the string manipulation that I am doing in getURI is not working. The values in inputTask.result are not being substituted by the value in Path.

This is the input and output to the Input Step

{
  "op1": "test1",
  "op2": "test2"
}

Output
{
  "op1": "test1",
  "op2": "test2"
  "responsePath": "$.op1/$.op2/response"
}

Is it possible to do some string manipulations using parameters in the Path in Step Function definition? If yes, what am I missing?

Thanks for your help.

2 Answers

You can use one or more EvaluateExpression Tasks - it's still a bit clunky.

You can find examples here.

API doc here.

Use a Lambda function instead of a Pass state to build the string.

Step Functions doesn't currently support string concatenation with reference paths. The Result field of a Pass state doesn't allow reference paths either. It has to be static value.

Related