Filling missing weekend rows with previous working day values

Viewed 115

I have a data table like the below. For each customer, missing days(weekends or holidays) should be inserted with the balance of previous working day. And this should only be done between the dates that customer has in the table. Balance should be added as 0 for dates outside the customer date range in the table. So for customer with id 1 should be filled between 2022-07-01 and 2022-07-31. Customer with id 2 should be filled between 2022-07-07 and 2022-07-19. Also for the dates 2022-07-01 to 2022-07-07 and 2022-07-19 to 2022-07-31 balance should be added as 0.

Data Table

date        customer_id      balance
2022-07-01      1            100
2022-07-04      1            150
2022-07-05      1            200
     .          1             .
     .          1             .
2022-07-31      1            650
2022-07-07      2            200
2022-07-08      2            300
2022-07-11      2            400
     .          2             .
     .          2             .
2022-07-19      2            750

Output table should look like this:

 date       customer_id      balance
 2022-07-01      1           100
 2022-07-02      1           100
 2022-07-03      1           100
 2022-07-04      1           150
 2022-07-05      1           200
      .          1            .
      .          1            .
 2022-07-31      1           650
 2022-07-01      2            0
 2022-07-02      2            0
      .          2            .
      .          2            .
 2022-07-07      2           200
 2022-07-08      2           300
 2022-07-09      2           300
 2022-07-10      2           300
 2022-07-11      2           400
      .          2            .
      .          2            .
 2022-07-19      2           750
 2022-07-20      2            0
      .          2            .
      .          2            .
 2022-07-31      2            0

There are some solutions that use cross join with calendar table to similar questions on the site. But i couldn't implement them for my case. Any help is much appreciated.

1 Answers

The below is a solution that uses recursion instead of a calendar table.

It essentially works by 'extending' your original data to create some extra rows with 0 balances for every customer at:

  1. The min date in the table (if the customer didn't already have a record at the min date)
  2. The max date in the table (if the customer didn't already have a record at the max date)
  3. The day after the last record for the customer (as long as this doesn't go over the max date in the table)

It then uses recursion to plug the gaps between the dates for each customer.

With balances as (
    -- This is a simplified version of the data already in your table
    SELECT '2022-07-01' as dt, 1 as customer_id, 100 as balance
    UNION ALL SELECT '2022-07-04' as dt, 1 as customer_id, 150 as balance
    UNION ALL SELECT '2022-07-05' as dt, 1 as customer_id, 200 as balance
    UNION ALL SELECT '2022-07-31' as dt, 1 as customer_id, 650 as balance
    UNION ALL SELECT '2022-07-07' as dt, 2 as customer_id, 200 as balance
    UNION ALL SELECT '2022-07-08' as dt, 2 as customer_id, 300 as balance
    UNION ALL SELECT '2022-07-11' as dt, 2 as customer_id, 400 as balance
    UNION ALL SELECT '2022-07-19' as dt, 2 as customer_id, 750 as balance
) 
, min_records as (
    -- This can create a 0 balance record for each customer at the min date in the table
    SELECT dt, customer_id, 0 as balance
    FROM (
        SELECT min(dt) as dt
        FROM balances
    ) as min_dt
    CROSS JOIN (
        SELECT DISTINCT customer_id
        FROM balances
    ) as customers
)
, max_records as (
    -- This can create a 0 balance record for each customer at the max date in the table
    SELECT dt, customer_id, 0 as balance
    FROM (
        SELECT max(dt) as dt
        FROM balances
    ) as min_dt
    CROSS JOIN (
        SELECT DISTINCT customer_id
        FROM balances
    ) as customers
)
, max_customer_records as (
    -- This creates a 0 balance record for each customer for the day after their last record,
    -- so long as that date does not go beyond the max date in the table
    SELECT dateadd(day, 1, max(dt)) as dt, customer_id, 0 as balance
    FROM balances as a
    CROSS JOIN (
        SELECT max(dt) as max_dt
        FROM balances
    ) as m
    GROUP BY customer_id, max_dt
    HAVING max(dt) < max_dt
)
, extended_balances as (
    -- We then join all of the tables above to the original balances table.
    -- Grouping to the dt + customer level and sum(balance) wont cause issues for customers
    -- who already had a record on the min(dt) or max(dt) because x + 0 still = x
    SELECT dt, customer_id, sum(balance) as balance
    FROM (
        SELECT * 
        FROM balances
        UNION 
        SELECT dt, customer_id, balance
        FROM min_records
        UNION 
        SELECT dt, customer_id, balance
        FROM max_records
        UNION 
        SELECT dt, customer_id, balance
        FROM max_customer_records
    ) AS A
    GROUP BY dt, customer_id
)
, recursive_query as (
    -- Now we use recursion to fill in the gaps between the dates
    SELECT dt as original_dt
        , dt
        , customer_id
        , balance
        -- We use lead() to find the date when a new balance exists
        , coalesce(lead(dt) over(partition by customer_id order by dt asc), dateadd(day, 1, dt)) as next_dt
    FROM extended_balances
    UNION ALL
    SELECT original_dt
        , dateadd(day, 1, dt)
        , customer_id
        , balance
        , next_dt
    FROM recursive_query 
    WHERE dateadd(day, 1, dt) < next_dt 
)

SELECT dt, customer_id, balance
FROM recursive_query
ORDER BY customer_id, dt

To help illustrate the steps, I've included examples of key tables:

Balances:

dt customer_id balance
2022-07-01 1 100
2022-07-04 1 150
2022-07-05 1 200
2022-07-31 1 650
2022-07-07 2 200
2022-07-08 2 300
2022-07-11 2 400
2022-07-19 2 750

Extended Balances:

dt customer_id balance
2022-07-01 1 100
2022-07-04 1 150
2022-07-05 1 200
2022-07-31 1 650
2022-07-01 2 0
2022-07-07 2 200
2022-07-08 2 300
2022-07-11 2 400
2022-07-19 2 750
2022-07-20 2 0
2022-07-31 2 0

First 10 records of the recursive query:

original_dt dt customer_id balance next_dt
2022-07-01 2022-07-01 1 100 2022-07-04
2022-07-01 2022-07-02 1 100 2022-07-04
2022-07-01 2022-07-03 1 100 2022-07-04
2022-07-04 2022-07-04 1 150 2022-07-05
2022-07-05 2022-07-05 1 200 2022-07-31
2022-07-05 2022-07-06 1 200 2022-07-31
2022-07-05 2022-07-07 1 200 2022-07-31
2022-07-05 2022-07-08 1 200 2022-07-31
2022-07-05 2022-07-09 1 200 2022-07-31
2022-07-05 2022-07-10 1 200 2022-07-31
Related