How to differentiate logs that don't have an ID to aggregate them separately on SQL?

Viewed 43

I have a (hypotetical) device that logs consumption events of its batteries. It has a few slots and daily logs the percentage consumed for the battery in each slot. This is what the data looks like:

CREATE TABLE batteries
(slot integer, day date, percentage integer);

INSERT INTO batteries
(slot, day, percentage)
VALUES
(0, '2020-05-08', 96),
(0, '2020-05-09', 96),
(0, '2020-05-10', 97),
(0, '2020-05-11', 97),
(0, '2020-05-12', 97),
(0, '2020-05-13', null),
(0, '2020-05-14', 95),
(0, '2020-05-15', 96),
(0, '2020-05-16', null),
(0, '2020-05-17', 1),
(0, '2020-05-18', 2),
(1, '2020-05-08', 10),
(1, '2020-05-09', 10),
(1, '2020-05-10', 10);

The log shows that in slot 0, a battery that was almost completely consumed has been replaced on May 13 by another used battery, which was then replaced on May 16 by a new one. The battery in slot 1 always reported 10% usage.

I need to identify each individual battery, the last value it has reported and the first and last dates it has reported such a value. So this is the output I'm trying to get:

slot  min_date      max_date      percentage  sequence
------------------------------------------------------
0     '2020-05-10'  '2020-05-12'  97          0
0     '2020-05-15'  '2020-05-15'  96          1
0     '2020-05-18'  '2020-05-18'  2           2
1     '2020-05-08'  '2020-05-10'  10          0

Last known value of battery 0 in slot 0 was 97, which it reported from May 10 to May 12;

Last known value of battery 1 in slot 0 was 96, which it reported only on May 15;

Last known value of battery 2 (current) in slot 0 was 2, which it reported only on May 18;

Last known value of battery 0 (current) in slot 1 was 10, which it reported from May 08 to May 10.

My main issue here is how to get the min and max date for each battery without having a battery id. In this example, if I group by slot and percentage to get the dates, I'll get the wrong min date on battery 1 of slot 0, because there was another battery with the same percentage in that slot before.

Is there a way I can get this result on a SQL query, without post-processing?

3 Answers

I have put some logic as following, which may (or may not) help you to get min and max date. check demo here.

with cte as
(
  select
    *,
    dense_rank() over (partition by percentage order by rnk desc) as nrnk
  from
  (
    select
          *,
          (day - '2000-01-01'::date 
               - row_number() over (partition by percentage order by day)) as rnk
      from batteries
      where percentage is not null  
  ) t   
) 

select
  slot,
  min(day) as min_day,
  max(day) as max_day,
  percentage
from cte
where nrnk = 1    
group by
  slot, 
  percentage, 
  nrnk

Output:

| slot | min_day    | max_day    | percentage |
| ---- | ---------- | ---------- | ---------- |
| 0    | 2020-05-17 | 2020-05-17 | 1          |
| 0    | 2020-05-18 | 2020-05-18 | 2          |
| 0    | 2020-05-14 | 2020-05-14 | 95         |
| 0    | 2020-05-15 | 2020-05-15 | 96         |
| 0    | 2020-05-10 | 2020-05-12 | 97         |
| 1    | 2020-05-08 | 2020-05-10 | 10         |

If I understand the data correctly, you know there is a new battery when the value is NULL. If that is the indication, then you can calculate the sequence by calculating the number of NULL values up to each row (using a cumulative sum).

You have one additional step to get the last value and then aggregate:

select slot,
       min(day) filter (where percentage = last_percentage),
       max(day), last_percentage,
       sequence
from (select b.*,
             first_value(percentage) over (partition by slot, sequence order by day desc) as last_percentage
      from (select b.*,
                   count(*) filter (where percentage is null) over (partition by slot order by day) as sequence
            from batteries b
           ) b
       where percentage is not null
     ) b
group by slot, sequence, last_percentage
order by slot, sequence;

Here is a db<>fiddle.

In Redshift you just use case expressions or booleans:

select slot,
       min(case when percentage = last_percentage then day end),
       max(day), last_percentage,
       sequence
from (select b.*,
             first_value(percentage) over (partition by slot, sequence order by day desc) as last_percentage
      from (select b.*,
                   sum( (percentage is null)::int ) over (partition by slot order by day) as sequence
            from batteries b
           ) b
       where percentage is not null
     ) b
group by slot, sequence, last_percentage
order by slot, sequence;

This is an iterative process. If you want to do this in SQL, use a recursive query. In order to do so, number your rows per slot first, so as to get easy from one row to the next one

In the recursive part:

  1. Compare a rows percentage with the one before and remember thus since when this percentage is current.
  2. Whenever you hit a null percentage you get a new battery (increase the battery number).

At last: Show the row with the highest date per slot and battery.

with recursive numbered as
(
  select
    slot, day, percentage,
    row_number() over (partition by slot order by day) as rn
  from batteries
)
, cte(slot, battery, min_date, max_date, percentage, rn) as
(
  select slot, 1, day, day, percentage, rn
  from numbered
  where rn = 1
   union all
  select
    n.slot,
    cte.battery + case when n.percentage is null then 1 else 0 end,
    case when n.percentage = cte.percentage then cte.min_date else n.day end,
    n.day,
    n.percentage,
    n.rn
  from cte
  join numbered n on n.slot = cte.slot and n.rn = cte.rn + 1
)
select distinct on (slot, battery)
  slot, min_date, max_date, percentage, battery - 1 as sequence
from cte
order by slot, battery, max_date desc;

Result:

+------+------------+------------+------------+----------+
| slot |  min_date  |  max_date  | percentage | sequence |
+------+------------+------------+------------+----------+
|    0 | 2020-05-10 | 2020-05-12 |         97 |        0 |
|    0 | 2020-05-15 | 2020-05-15 |         96 |        1 |
|    0 | 2020-05-18 | 2020-05-18 |          2 |        2 |
|    1 | 2020-05-08 | 2020-05-10 |         10 |        0 |
+------+------------+------------+------------+----------+

Demo: https://dbfiddle.uk/?rdbms=postgres_12&fiddle=1ee3278d6f020f2d1a61bc93276965f5

Related