I'm using Spinnaker on Google Kubernetes Engine with Stackdriver as a log and statistics source. I'm having trouble getting timing/distribution metrics working with the Spinnaker automated canary analysis setup.
My application logs request information for each API request coming in. They include all the request details so I can create log-based stats.
A fairly truncated log entry looks like this:
{
"jsonPayload": {
"format_parameters": {
"ElapsedMilliseconds": "4.9541",
"Method": "POST",
"NormalizedPath": "api/add",
"StatusCode": "200"
}
},
"labels": {
"kubernetes/labels/app.kubernetes.io/name": "calculator",
"kubernetes/labels/track": "production",
"kubernetes/namespace": "calculator"
}
}
I've created a log-based statistic for request latency using a log filter like this:
resource.type="container" AND
jsonPayload.format_parameters.NormalizedPath != ""
And specifying it's a "Distribution" metric based on jsonPayload.format_parameters.ElapsedMilliseconds. I've called it api/request_latency and I'm able to see graphs of the data, break it down by path and request method, etc.
I want to set up Spinnaker (Kayenta) so it will fail the canary analysis if latency increases. This article from Netflix shows they're doing that but I also see they're judging it by 5th, 50th, and 99th percentiles.
I've set up my canary configuration to look at logging.googleapis.com/user/api/request_latency and I've got a filter like this:
metric.label.name="${location}" AND
metric.label.track="${scope}"
The track label switches between canary and production and the location is set up as the app name.
On executing the canary analysis, I get this error:
Field aggregation.perSeriesAligner had an invalid value of "ALIGN_MEAN": The aligner cannot be applied to metrics with kind DELTA and value type DISTRIBUTION."
I've found this is coming from Stackdriver and the Kayenta query running against the statistics has ALIGN_MEAN hardcoded as the per-series aligner.
I'm not a stats guy, and I'm honestly somewhat overwhelmed by having to figure out what things like "per-series aligner" even means.
How do I get request latency out of my logs and use that in canary analysis?