How to include google benchmark in continuous integration

user_2416647 0
Viewed 2183

I like to include google benchmark tests within out continuous integration environment.

How can I check if the code under test has not become slower than it was before? Do I have to manually process the json output of the benchmark?

Is there an elegant way of accounting for different machines?

Is there an elegant way to check for every build on the same machine the performance? Is there a better way than writing a bash script and compare the output of the actual performance with a reference performance?

There has been a similar question, but no good answer to it.

Is there a way to integrate it into Jenkins.

continuous jenkins integration benchmarking c++
2 Answers

To integrate Google Benchmark in Jenkins, I used this neat plugin:

https://plugins.jenkins.io/benchmark

I had the tests output .json files

benchmarks.exe --benchmark_out=benchmarktest_output.json

and wrote a custom JSON schema for the plugin to understand the output:

{
  "description": "Google Benchmark JSON schema",
  "failure": { "value": true },
  "type": "object",
  "properties": {
    "benchmarks": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "name": { "type": "name" },
          "real_time": { "type": "result" },
          "cpu_time": { "type": "result" },
          "iterations": { "type": "parameter" }
        }
      }
    }
  }
}

I'm not able to display all the output in the plugin though. E.g., I haven't figured out how to display the time unit together with both real and cpu time, only with one of them at the time.

user_56033310

With this python tool one can automate the evaluation of google benchmark tests and thus integrate them into a continuous integration tool

Thanks you @MikeVanDyke for the hint.

user_24166470
Related