Azure Stream Analytics: When does a stream analytics job actually process data if the job query is a day wise TUMBLINGWINDOW?

user_14599889 0
Viewed 166

Context

I have created a streaming job using Azure portal which aggregates data using a day wise TUMBLINGWINDOW. Have attached a code snippet below, modified from the docs, which shows similar logic.

SELECT
    DATEADD(day, -1, System.Timestamp()) AS WindowStart
    System.Timestamp() AS WindowEnd, 
    TollId, 
    COUNT(*)
FROM Input TIMESTAMP BY EntryTime  
GROUP BY TumblingWindow(day, 1), TollId

Question

If the TUMBLINGWINDOW outputs at the end of the window (which in the case that I start my job at midnight of any given day would mean shortly after midnight of the next day) then during the day is data still being processed or does the processing only happen based on when the query is outputting?

An explanation in detail as to how this would work would be great. Haven't found any documentation which really explains these concepts in detail (with these edge cases)

Thoughts

I am trying to gauge how if I stop a job from running and start it again from "When Last Stopped" would it still lead to the same aggregation as if I'd left it on all the time (if it would then how)? Bearing in mind I am using a day wise TUMBLINGWINDOW?

azure-stream-analytics azure tsql
1 Answers

The output time of the tumbling window is absolute, and not dependent on the query start time. A daily tumbling window generates an output at 00:00:00AM, an hourly one every top of the hour (00:00:00AM, 01:00:00AM...), etc.

So here the job is waiting for 24h, loading patiently the data in memory, until it's 00:00AM so it can perform the computation and output the results. Then it starts waiting again.

Here, with a daily window, nothing prevents you from stopping the job from 00:01AM to 23:59PM.

(EDIT - THIS IS NOT CORRECT - FIXED BELOW) Just be mindful that when you start it, the start time option needs to cover the missing time (so either 'when last started' - because we checkpoint data - or custom time 24h before).

(CORRECTION) Just be mindful that when you start it, the start time option needs to cover the output window you want covered - ASA will reload all the necessary data even if it's before that time. What you drive with the start time is the output time, not the data input period.

As long as the data is still here (be mindful of the retention period of the event hub, by default 1 day), you could pause for an entire week, and have the job reprocess the whole period to emit 7 results. For that you just need a start time that covers the period.

Note that it takes time to re-ingest the whole dataset, and compute the operation on it. So if you absolutely need your daily average to be out at 00:00:00AM, then restart the job a few minutes prior so it can catch up. Else you will get that output at 00:00:10AM (or whatever the times it take to reload the data in memory).

user_6339250
Related