Moving (Rolling) Median with BigQuery

user_10556104 0
Viewed 805

I currently have a table in BigQuery that contains some outliers and would like to calculate the moving median for this table.

Example table:

port - qty - datetime
--------------------------------
TCP1 - 13 - 2018/06/11 11:20:23
UDP2 - 15 - 2018/06/11 11:24:24
TCP3 - 12 - 2018/06/11 11:24:27
TCP1 - 2  - 2018/06/12 11:24:26 
UDP2 - 15 - 2018/06/12 11:35:32
TCP3 - 200- 2018/06/13 11:45:23
TCP3 - 14 - 2018/06/13 11:54:22
TCP3 - 13 - 2018/06/14 11:55:33
TCP1 - 17 - 2018/06/15 11:43:33
UDP2 - 12 - 2018/06/15 11:55:25
TCP3 - 14 - 2018/06/15 11:26:21
TCP3 - 11 - 2018/06/16 11:55:46
TCP1 - 14 - 2018/06/17 11:34:33
UDP2 - 15 - 2018/06/17 11:43:24
TCP3 - 13 - 2018/06/17 11:47:54
and ...

I would like to be able to calculate a 7-day moving median on the various ports at 11hrs using bigquery standard SQL. I have tried calculating the moving average, but have realised that the calculations are affected by the 'outlier'.

I do not know how to write the SQL query to calculate the moving median. Any help will be greatly appreciated.

(this is the closest thread that i could find on this topic: BigQuery - Moving median calculation but I need bigquery to be able to get the qty from the table as I do not know exactly the number of qty for each specific day)

bigquery-standard-sql google-bigquery median sql
1 Answers

I think this is close enough to what you want:

select t.*,
       qtys[ordinal(cast(array_length(qtys) / 2 as int64))]
from (select t.*,
             array_agg(qty) over (partition by port
                                  order by datetime_diff(datetime, datetime('2000-01-01'), day)
                                  range between 7 preceding and current day
                                 ) as qtys
      from t
      where extract(hour from datetime) = 11
     ) t;

Medians are a bit tricky when there are an even number of rows in the result set. This chooses an arbitrary value.

user_11440350
Related