How to configure AWS StepFunction flow based on the result of the previous step?

user_10581944 0
Viewed 122

I have a StepFunction, first step is to run some Python code in Glue, then generate some data and stored in S3 bucket, then the next step is to look at this file and run a Lambda function.

I want to print out in the Glue python code, to see if the generated file is empty or not, if not empty, continue the nest step. It it's empty, skip the rest of the steps and StepFunction succeeded. My question is that it there any examples how to achieve this? Or is it even possible to decide StepFunction flow like this? (All the AWS resources are managed by Terraform) Thanks.

aws-step-functions aws-glue terraform amazon-web-services python
1 Answers

Yes, you could very well do this. Try an approach (flowchart and not actual stepfunction) as shown below: enter image description here

The first Lambda function will run the glue job for you. Make sure the state machine knows the output location. Ideally, passing the required data to the Glue Job via the invoking lambda function.

The second lambda function will check if the file has data and return a status, say for example {'data-availability-status': True} or {'data-availability-status': False}

The third state machine is a Choice state that act's on 'data-availability-status' from Check File state. If the status is True, it should go to next steps and if the status is False it should gracefully exit.

user_27388090
Related