I have two aerospike clusters running community editions -
Older cluster with the servers having the following combinations - storing data in disk + i2.2xlarge instances + Aerospike build vesion 3.8.2.3
Newer cluster with the servers having the following combinations - storing data in memory + r3.4xlarge instances + Aerospike build version 3.14.1.1 + using partition-tree-sprigs
I wanted to compare the server side latencies and timeouts on them. I enabled the asgraphite daemon which comes in-built with Aerospike, by the following command -
python /opt/aerospike/bin/asgraphite --start --prefix aerospike.stats -g <URL> -p <port>
I cannot see the latency stats against the old cluster in the graphite console (see highlight in screenshot) -
Also, I am confused, as to which latency stat should I consider. Following stats are available against the older cluster -
Metric Value observed on one node
batch_index_timeout 0
batch_timeout 0
err_tsvc_requests_timeout ~80K
stat_rw_timeout ~500K
The batch stats are showing 0, as expected, because we are not doing any batch queries. The new cluster being higher than 3.9 does not have the err_tsvc_requests_timeout and stat_rw_timeout metrics.
The relevant manual page of aerospike mentions against the deprecated metrics -
As of version 3.9, refer to the more specific stats at the namespace level.
Not sure which are those.
Opening Bounty
The metrics reference manual mentions against metricstat_rw_timeout -
As of version 3.9, refer to the more specific stats at the namespace level.
I am expecting the same to be reflected against the namespace value in the graphite web console, but all I can see are the following - ops_per_sec, over_1ms, over_64ms etc.
So, basically I am looking for two things now -
Exact meaning of some stat/metric getting moved to namespace level. How can it be viewed in graphite web console, it at all it can be seen.
More pointers towards selecting the appropriate metrics for latency and timeout across both the versions. I am working on the common use case of reading and writing aerospike cache keys via the PHP client API functions -
Aerospike->get() Aerospike->put()
Update 2
timeouts - Finally, I am able to locate the reorganized timeout related parameters in the new build version, as explained in the answer https://stackoverflow.com/a/45244090/351903. But, the values client_write_timeout etc being cumulative, are difficult to compare across clusters, as logging might have started earlier in one of the clusters. It would have been better to have an instantaneous metric for timeouts.
Update 3
latency - Since I did not get latency stats in graphite web console for build version less than 3.9, I am planning to use asloglatency and dump the stats to graphite server for both the build versions. To uniformly compare latencies in a graphical way, I have planned to -
Setup a cron which runs every 5 mins.
Run the asloglatency command to collect the following latency stats for 2 minutes duration, starting from last 5 minutes for both the build versions -
avg of - % greater than 1 ms, 8 and 64 ms and ops per second.
maximum of - % greater than 1 ms, 8 and 64 ms and ops per second.
Asloglatency command for Version > 3.9
asloglatency -N FC -h write -f -0:05:00 -d 0:60:00
asloglatency -N FC -h read -f -0:05:00 -d 0:60:00
Command for Version < 3.9
asloglatency -h writes_master -f -0:05:00 -d 2:00
asloglatency -h reads -f -0:05:00 -d 2:00
Note - I got latency stats for new build by running asgraphite command -
python /opt/aerospike/bin/asgraphite --start -g <domain> -p <port>
But I am not sure which of the above stats are logged in case of new build under {HOSTNAME}.latency in graphite console - avg or max values. I did not find any documentation of the same in the metrics reference manual.
Also, the above command did not show latency stats in graphite console for older build.
Hopefully, the stats derived using asloglatency will be uniform across the two build versions.
Via open bounty - Looking for confirmations that this would work/would be the best way to do what I am trying to find out/pointers to easier ways of doing the same.
Update 4
1. Timeout
I am able to get instantaneous timeouts on old build by applying derivative() on the graph obtained by logging stat_rw_timeout -
http://<domain>/render?width=1700&from=-6h&until=now&height=900&target=derivative(aerospike.old_statsip-10-146-210-31.service.stat_rw_timeout)&title=old_latency_cumulative_derivative&hideLegend=false&_salt=1367201670.479&yMax=&_uniq=0.985620767407021
However, timeouts on new build are showing 0 consistently even though client_read_success is showing values -
2. Latency
Looks like the following asgraphite command logs latency in both the build versions, so I will not need to do all those stuff using asloglatency as I mentioned in my Update 3 -
python /opt/aerospike/bin/asgraphite -l 'latency:' --start --prefix aerospike.temp.old_trial1 -g <graphite server domain> -p <port>
The values I am monitoring are -
New build -
aerospike.temp.new_trial2ip-10-13-215-20.latency.FC.read.over_1ms
aerospike.temp.new_trial2ip-10-13-215-20.latency.FC.write.over_1ms
Old build -
aerospike.temp.old_trial1ip-10-182-71-216.latency.reads.over_1ms
aerospike.temp.old_trial1ip-10-182-71-216.latency.writes_master.over_1ms
Here are the results observed from the graphs -
- Read latency on new build - consistently 0.
- Write latency on new build - only one spike, otherwise consistently 0.
- Read and write latency on old build - consistent data.
Update 5
I just want to make I am comparing the right metrics for latency and timeouts comparison across the build versions. Can someone point me to some documentation related to the same?
Latency - I have mentioned the values I am comparing in update 4 above. Following is their hierarchy in the graphite web console -
Timeouts - There is no direct mention in the metrics reference manual that stat_rw_timeout splits into client_read_timeout and client_write_timeout in version 3.9. Can someone confirm the same?
I am having above concerns/queries because of the following conclusions from my observations -
