My question is about the CancellationToken passed into C# functions: https://docs.microsoft.com/en-us/azure/azure-functions/functions-dotnet-class-library#cancellation-tokens
We have observed that occasionally when we are in the middle of running a function, the CancellationToken is raised and causes all our async calls to stop. If a CancellationToken is signaled, we do expect this behavior, but I have a more general question about what causes this CancellationToken to be signaled in the first place. According to the documentation, I am left with a sort of vague feeling of understanding on the topic.
It seems like it just can happen any time the operating system says "it is time to shut down what we are working on". But I have no idea under which conditions this can and will happen when hosting functions within Azure. Does this ever happen as a result of functions running on the consumption plan needs to "scale out" under the hood? If not, what exactly (in the context of hosting within Azure on the consumption plan) causes this to happen?
Further more, should we be passing the CancellationToken along to async calls? I feel like the answer is yes, but maybe this is our mistake in the first place. Maybe we aren't supposed to pass the CancellationToken along at all, but it would feel so strange.
For a little more context on how we came across this issue/question in the first place:
- We have a TimerTrigger function that runs once a day which queues tens of thousands of messages onto an Azure Storage Queue using the Azure.Storage.Queues nuget package.
- We have a QueueTrigger function that picks up messages from the queue and makes calls to an HTTP dependency.
What we observed is that our TimerTrigger function was failing often after a couple thousand messages, with exceptions along the lines of TaskCanceledException. My first guess was that the function was timing out after 10 minutes, but it usually happens only a minute into execution.
My working theory is that after queuing messages begins, the QueueTrigger function begins to process them with high levels of parallelization. However, after a certain threshold the underlying infrastructure (ie: the consumption plan VMs) determine that in order to handle the load it should scale out. If this is the case, maybe the functions that were started on the original machine (ie, the TimerTrigger function) are signaled to wrap up what they are doing via the CancellationToken since the VM is being free'ed up/abandoned.
I realize that the internal workings of Azure are not something that are typically discussed in regards to how functions are scaled, etc. But I would really like to know what exactly is causing my TimerTrigger's CancellationToken to be signaled.
For anyone that runs across this issue and wonders how we ended up solving at (at least in the short term): We decided to take our logic in the TimerTrigger and refactor into a durable function. So basically the TimerTrigger simply kicks off an OrchestrationTrigger and we have an ActivityTrigger that queues a much smaller number of messages at a time. We call that ActivityTrigger from the OrchestrationTrigger with a retry policy, which allows the process to complete gracefully even when a CancellationToken is signaled in the ActivityTrigger function (which does happen about 10-15% of the time, from my initial testing).