On my Prometheus instance, I have set storage.tsdb.retention.size to 128GiB and storage.tsdb.retention.time to 0s, so Prometheus keeps the old data until the 128 GB limit is reached.
Now, I have some time series that haven't been updated anymore (i.e. are stale) for a long time. If I make a range query into the recent past where the stale metrics didn't exist anymore, then everything is fine.
For example, the PromQL query:
> metric{label1="foo"}[1d]
Returns:
...
metric{label1="foo"} <value>@<timestamp> # <== OK, fresh time series
However, if I make a range query reaching farther into the past when the stale metric was still updated, this stale time series is included in the result.
For example, the query:
> metric{label1="foo"}[60d]
Returns:
...
metric{label1="foo"} <value>@<timestamp> # <== OK, newest timestamp right now
...
metric{label1="foo",label2="bar"} <value>@<timestamp> # <== !! newest timestamp one month ago!
I don't want the second (stale) time series included in the result, I only want the data from first (fresh) time series reaching 60 days back.
Is there a way to achieve this with PromQL, i.e. filtering out stale time series from range queries?