How to use InfluxDB to track how weather forecast changed up to forecast date

user_27524 0
Viewed 49

I'm trying to get familiar with InfluxDB and time series databases in general and wonder if this is an appropriate usecase.

Thinking about weather forecasts up to the forecasted date. Let's say daily you have a max temperature prediction for the next 5 days, so for every actual date you have 5 forecasted max temp values followed by the actual value.

So the timestamp would be the date a prediction is made, but what do you then use for the time that the prediction is for? I would think that in this case the prediction might be for a given date, but it could also be for a subset of a date. Would this be a tag?

influxdb
1 Answers

I had the same issue. I solved it with timestamp and tags:

  1. timestamp: put the forecast values on the timestamp of the time they are predicted for.
  2. add to each value a tag 'age_h' (age of the forecast in hours in my case) I know, tags can only be strings but I can still add a number as string. The age in this case is actually the age into the future. So if today you add a forecast for 2 days later, the age_h tag becomes 48. If you ever look at that point again in the future, you know that the value was predicted 48h before the actual timestamp of the value. You will get a series of values for the same timestamp but they will have different prediction age_h and different accuracy probabilities according to their age. If you search the most recent prediction, go for the value with the lowest age_h number. (can that be done in flux? I don't know how. Yet.)
  3. Dont do this, see Edit below add to each value a tag 'forecast_series' giving the time, the forecast was made. Again, only string allowed, but it doesn't stop me from adding it. <year><month><day><hour>(I had delimeters first but then realized that it will be easier to compare if I can parse it as number). This tag ensures, that you can find the forecast/prediction series in case you need that dimension.

I haven't got a lot of data yet, so I need to see, whether it works out as expected.

How did you solve it?

Edit: Apparently the third tag was a bad idea. Suddenly my free online buckets didn't want to take any more data. With the following Error:

Oh no! You hit the series cardinality limit and your data stopped writing.

The value range of tags should not increase steadily. The range of used tags and their values should be limited to keep the DB fast and slim.

So I figured out that the third tag can actually be deducted from the timestamp and age_h tag.

This is now a pretty neat query for showing the graphs with all values with tag age_h == '0' for past values and the most recent forecasts for the future values.

If you want more details, please leave a comment. If it can be improved, please enlighten me :-)

import "strings"
import "system"
import "date"
import "timezone"

hours = (h) => duration(v:(h)*1000000000*3600)

to_int_hours = (t) => {
  datestr = string(v:t)
  year = strings.substring(v: datestr, start: 0, end: 4)
  mon = strings.substring(v: datestr, start: 5, end: 7)
  day = strings.substring(v: datestr, start: 8, end: 10)
  hour = strings.substring(v: datestr, start: 11, end: 13)
  return int(v: year+mon+day+hour)
}

forecast_datetime = (t, z, a) => { //time, zone_h, age_h_string
  a_h = int(v: a)
  forecast_date = date.add(d: hours(h:z-a_h), to: t)
  return to_int_hours(t:forecast_date)
}

local_zone = 2 //my zone: UTC+2
local_now = date.add(d: hours(h:local_zone-1), to: system.time())
query_datetime = to_int_hours(t:local_now)

rangeStop = date.add(d: 48h, to: v.timeRangeStop)

from(bucket: "Energy")
  |> range(start: v.timeRangeStart, stop: rangeStop)
  |> filter(fn: (r) => r._measurement == "meteo-forecast")
  |> filter(fn: (r) => r._field == "TTT_C"  or r._field == "FF_KMH")
  |> filter(fn: (r) => r.age_h == "0")
  |> drop(columns: ["age_h"])
  |> sort(columns: ["_time"])
  |> yield(name: "temperature_wind_past")
from(bucket: "Energy")
  |> range(start: v.timeRangeStart, stop: rangeStop)
  |> filter(fn: (r) => r._measurement == "meteo-forecast")
  |> filter(fn: (r) => r._field == "TTT_C" or r._field == "FF_KMH")
  |> filter(fn: (r) => forecast_datetime(t:r._time, z:local_zone, a:r.age_h) >= query_datetime)
  |> drop(columns: ["age_h"])
  |> sort(columns: ["_time"])
  |> yield(name: "temperature_wind_forecast")
from(bucket: "Energy")
user_107588050
Related