What's the most efficient way to calculate a rolling aggregate in SQL?

Viewed 610

I have a dataset that includes a bunch of clients and date ranges that they had a "stay." For example:

| ClientID | DateStart | DateEnd |
+----------+-----------+---------+
| 1        | Jan 1     | Jan 31  | (datediff = 30)
| 1        | Apr 4     | May 4   | (datediff = 30)
| 2        | Jan 3     | Feb 27  | (datediff = 55)
| 3        | Jan 1     | Jan 7   | (datediff = 6)
| 3        | Jan 10    | Jan 17  | (datediff = 6)
| 3        | Jan 20    | Jan 27  | (datediff = 6)
| 3        | Feb 1     | Feb 7   | (datediff = 6)
| 3        | Feb 10    | Feb 17  | (datediff = 6)
| 3        | Feb 20    | Feb 27  | (datediff = 6)

My ultimate goal is to be able to identify the dates on which a client passed a threshold of N nights in the past X time. Let's say 30 days in the last 90 days. I also need to know when they pass out of the threshold. Use case: hotel stays and a VIP status.

  • In the example above, Client 1 passed the threshold on Jan 31 (had 30 nights in past 90 days), and still kept meeting the threshold until April 2 (now only 29 nights in the past 90 days), but passed the threshold again on May 4.
  • Client 2 passed the threshold on Feb 3, and kept meeting the threshold until April 28th, at which point the earliest days are more than 90 days ago and they expire.
  • Client 3 passed the threshold on around Feb 17

So I would like to generate a table like this:

| ClientID | VIPStart  | VIPEnd  |
+----------+-----------+---------+
| 1        | Jan 31    | Apr 2   |
| 1        | May 4     | Jul 5   |
| 2        | Feb 3     | Apr 28  |
| 3        | Feb 17    | Apr 11  |
(Forgive me if the dates are slightly off, I'm doing this in my head)

Ideally I would like to generate a view, as I will need to reference it often.

What I want to know is what's the most efficient way to generate this? Assuming I have thousands of clients and hundreds of thousands of stays.

The way that I've been approaching this so far has been to use a SQL statement that includes a parameter: as of {?Date}, who had VIP status and who didn't. I do that by calculating DATEADD(day,-90,{?Date}), then excluding the records that are out of the range, then truncating the DateStarts that extend earlier and DateEnds that extend later, then calculating the DATEDIFF(day,DateStart,DateEnd) for the resulting stays using adjusted DateStart and DateEnd, then getting a SUM() of the resulting DATEDIFF() for each Client as of {?Date}. It works, but it's not pretty. And it gives me a point in time snapshot; I want the history.

it seems a little inefficient to generate a table of dates and then for every single date, use the above method.

Another option I considered was converting the raw data into an exploded table with each record corresponding to one night, then I can count it easier. Like this:

| ClientID | StayDate  |
+----------+-----------+
| 1        | Jan 1     |
| 1        | Jan 2     |
| 1        | Jan 3     |
| 1        | Jan 4     |
etc.

Then I could just add a column counting the number of days in the past 90 days, and that'll get me most of the way there.

But I'm not sure how to do that in a view. I have a code snippet that does this:

WITH DaysTally AS (
    SELECT MAX(DATEDIFF(day, DateStart, DateEnd)) - 1 AS Tally
    FROM Stays
UNION ALL
    SELECT Tally - 1 AS Expr1
    FROM DaysTally AS DaysTally_1
    WHERE  (Tally - 1 >= 0))

SELECT t.ClientID,
        DATEADD(day, c.Tally, t.DateStart) AS "StayDate"
    FROM Stays AS t 
    INNER JOIN DaysTally AS c ON 
        DATEDIFF(day, t.DateStart, t.DateEnd) - 1 >= c.Tally

OPTION (MAXRECURSION 0)

But I can't get it to work without the MAXRECURSION and I don't think you can save a view with MAXRECURSION

And now I'm rambling. So the help that I'm looking for is: what is the most efficient method to pursue my goal? And if you have a code example, that would be helpful too! Thanks.

2 Answers

This is an interesting and pretty well-asked question. I would start by enumerating the days from the beginning of the first stay of each client until 90 days after the end of its last stay with a recursive cte. You can then bring the stay table with a left join, and use window functions to flag the "VIP" days (note that this assumes no overlaping stays for a given client, which is consistent with your sample data).

What follows is gaps-and-islands: you can use a window sum to put "adjacent" VIP days in groups, and then aggregate.

with cte as (
    select clientID, min(dateStart) dt, dateadd(day, 90, max(dateEnd)) dateMax
    from stays 
    group by clientID
    union all
    select clientID, dateadd(day, 1, dt), dateMax 
    from cte 
    where dt < dateMax
)
select clientID, min(dt) VIPStart, max(dt) VIPEnd
from (
    select t.*, sum(isNotVip) over(partition by clientID order by dt) grp
    from (
        select 
            c.clientID, 
            c.dt,
            case when count(s.clientID) over(
                partition by c.clientID 
                order by c.dt 
                rows between 90 preceding and current row
            ) >= 30
                then 0
                else 1
            end isNotVip
        from cte c
        left join stays s 
            on c.clientID = s.clientID and c.dt between s.dateStart and s.dateEnd
    ) t
) t
where isNotVip = 0
group by clientID, grp
order by clientID, VIPStart
option (maxrecursion 0)

This demo on DB Fiddle with your sample data produces:

clientID | VIPStart   | VIPEnd    
-------: | :--------- | :---------
       1 | 2020-01-30 | 2020-04-01
       1 | 2020-05-03 | 2020-07-04
       2 | 2020-02-01 | 2020-04-28
       3 | 2020-02-07 | 2020-04-20

You can put this in a view as follows:

  • the order by and option(maxrecursion) clauses must be omitted when creating the view

  • each and every query that has the view in its from clause must end with option(max recursion 0)

Demo

You can eliminate the recursion by creating a tally table in the view. The approach is then the following:

  • For each period, generate dates from 90 days before the period to 90 days after. These are all the "candidate days" that the period could affect.
  • For each row, add a flag as to whether it is in the period (as opposed to the 90 days before and after).
  • Aggregate by client id and date.
  • Use a running sum to get the days with 30+ in the previous 90 days.
  • Then filter for the ones with 30+ days and treat this as a gaps-and-islands problem.

Assuming that 1000 days is sufficient for the periods (including the 90 days before and after), then the query looks like this:

with n as (
      select v.n
      from (values (0), (1), (2), (3), (4), (5), (6), (7), (8), (9)) v(n)
     ),
     nums as (
      select (n1.n * 100 + n2.n * 10 + n3.n) as n
      from n n1 cross join n n2 cross join n n3
     ),
     running90 as (
      select clientid, dte, sum(in_period) over (partition by clientid order by dte rows between 89 preceding and current row) as running_90
      from (select t.clientid, dateadd(day, n.n - 90, datestart) as dte,
                   max(case when dateadd(day, n.n - 90, datestart) >= datestart and dateadd(day, n.n - 90, datestart) <= t.dateend then 1 else 0 end) as in_period
            from t join
                 nums n
                 on dateadd(day, n.n - 90, datestart) <= dateadd(day, 90, dateend)
            group by t.clientid, dateadd(day, n.n - 90, datestart)
           ) t
     )
select clientid, min(dte), max(dte)
from (select r.*,
             row_number() over (partition by clientid order by dte) as seqnum
      from running90 r
      where running_90 >= 30
     ) r
group by clientid, dateadd(day, - seqnum, dte);

Having no recursive CTE (although one could be used for n), this is not subject to the maxrecursion issue.

Here is a db<>fiddle.

The results are slightly different from your results. This is probably due to some slight difference in the definitions. The above includes the end day as an "occupied" day. The 90 days is 89 days before plus the current day in the above query. The second-to-last query shows the 90 days running days, and that seems correct to me.

Related