This can be considered a follow-up question to Invoking the same activity inside a loop in cadence workflow: How does the workflow recover in case of activity iteration ? Will is continue invoking the i-th activity (skipping the ones that were already invoked) or start again from 0 ? If so, how can the workflow be written so that activities that were invoked (0-'k') are skipped ?
3 Answers
First of all, it's important to understand the difference between workflow worker vs activity worker.
- Workflow worker only processes your workflow code (not inside the activities) and stops at Cadence API calls to schedule an activity, to start a timer etc. Workflow workers receive "decision tasks" where they execute the workflow code from last point it paused till the next API call. When sticky workers are used, there will be a thread waiting on the last API call maintaining the local variables. If the worker didn't already have such thread, it will replay the workflow using the history first to understand where the workflow was supposed to be waiting.
- Activity worker only executes "inside the activities" given activity parameters. Think of like a cloud function. Activity workers will receive activity tasks containing activity parameters, they will execute the activity and return its result. That's it.
Coming back to your question: When your "workflow" worker crashes, restarts, lost or somehow a worker received a decision task that it doesn't know about its workflow; it will then request the history of that workflow from Cadence servers. It will replay the workflow until is consumes the whole history it receives then will schedule the next activity based on current local variables.
So it will continue from the same loop iteration with the same local variables and parameters to the activity.
Hope this answers your question
My understanding is those iterations are replayed when the history is retrieved in the rescuing worker.
So iteration 1 had result 'x', iteration 2 had 'y' until it reaches the last activity that it hadn't recorded a result for, and then starts actually invoking the activities.
In this way, you can be sure the state of the workflow, and its potential variables, is exactly how it was when it terminated.
So they aren't exactly 'skipped' but they also don't re-execute.
If you don't want to replay hundreds of no-op iterations after a failure, you could consider using "continueAsNew" instead of a loop. This ensures the new workflow doesn't accrue the history of the other failed iterations.
Each activity invocation has its own retry policy, so if activity i fails, only activity i will be retried.
Related