I am triggering a Step Function execution via a Python cell in a SageMaker Notebook, like this:
state_machine_arn = 'arn:aws:states:us-west-1:1234567891:stateMachine:alexanderMyPackageStateMachineE3411O13-A1vQWERTP9q9'
sfn = boto3.client('stepfunctions')
..
sfn.start_execution(**kwargs) # Non Blocking Call
run_arn = response['executionArn']
print(f"Started run {run_name}. ARN is {run_arn}.")
and then in order to check that the execution (which might take hours to complete depending on the input) has been completed, before I start doing some custom post-analysis on the results, I manually execute a cell with:
response = sfn.list_executions(
stateMachineArn=state_machine_arn,
maxResults=1
)
print(response)
where I can see from the output the status of the execution, e.g. 'status': 'RUNNING'.
How can I automate this, i.e. trigger the Step Function and continue the execution on my post-analysis custom logic only after the execution has finished? Is there for example a blocking call to start the execution, or a callback method I could use?
I can think of putting a sleep method, so that the Python Notebook cell would periodically call list_executions() and check the status, and only when the execution is completed, continue to rest of the code. I can statistically determine the sleep period, but I was wondering if there is a simpler/more accurate way.
PS: Related: How to avoid simultaneous execution in aws step function, however I would like to avoid creating any new AWS resource, just for this, I would like to do everything from within the Notebook.
PPS: I cannot make any change to MyPackage and the Step Function definition.